Opinions sought on information presentation

Tue Apr 19 21:42:48 PDT 2011

While I'm not actually a sysadmin per se any more, there's a fair amount
of overlap between what I'm doing and what I did as a sysadmin.  And
it's possible that some of y;all might find some of it of interest,
so....

I recently was able to actually get some hardware that I'm allowed to
reconfigure, so I've been performing series of
as-identical-as-I-can-make-them runs of a sample workload that is of
"significant" interest to my employer.

In the process, I've been gathering information during the workload in
question:  I cobbled up a Perl script that runs a command under the
auspices of "/usr/bin/time -lp", parses the resulting output, then spits
out that information, together with just about anything else I was able
to think of that might be interesting, all in a simple "tagged" format
that's fairly easy to parse while remaining flexible (so I can change
the mix of stuff reported -- or the sequence of things -- with
impunity).

But before it does that, it forks a little shell script (and when the
command under examination terminates, the Perl script kills the shell
script).  The shell script sits mostly in the background, and
periodically (default: every 10 seconds) grabs a handful of
readily-available information (mostly from sysctl(8); stuff from
"netstat -nibf inet" is included by default) and re-formats that to the
same "tagged" format (but appends it to a different file) byeans of
awk(1).

Among the data thus retrieved are the 1-, 5-, and 15-minute load
averages; another set that can be interesting is an ordered quintuple:
CPU utilization counters for each of the states "system", "interrupt",
"user", "nice", and "idle" (in that sequence).

These latter are what is used to determine CPU utilization (e.g., by
top(1) or vmstat(8)).

[It turns out that I actually can (and do) also get the per-core versions
of these counters.]

In any case, in an attempt to gain a bit more understanding of the
behavior of the system, I cobbled up a bit of code:

* Some Perl, to split the multivariate reported values (such as the CPU
  counters) into separate univariate values.

* Some Perl, to read the "tagged" data & spit out a "delimited" file
  with headers.

* Some R, to accept a "data frame" (result of having read output of that
  last) and generate a graph showing the distribution of CPU states over
  time during the execution of the command being examined.

Over the last couple of days, I hacked this last to also (optionally)
overlay the load averages on the same graph.

So I'm interested in opinions on the result.  I've placed a couple of
lightly-sanitized graphs up at <http://www.catwhisker.org/~david/perf/>,
and solicit comments.

(I'm aware that the legends are a bit truncated.  I also gather that
some folks may not readily distinguish between the read & the green
colors I used: the code I wrote permits those colors to be overridden,
but if more generally useful defaults make sense, I'd like to know.  I
don't pretend to be an expert in data presentation.)

Mostly, I'm interested in knowing if other folks perceive the way the
information is presented as (arguably) useful and fairly clear.

I haven't generated any of the per-core graphs for this exercise; if
there's interest, I could probably be persuaded to do so.

Thanks!

[Oh: all of this is done in a FreeBSD environment.  Each of the tools
used is open source.  I've indicated to my manager that I'd like to be
able to release what I've done so others might improve it.  She hasn't
actually raised any objection so far....]

Peace,
david
-- 
David H. Wolfskill				david at catwhisker.org
Depriving a girl or boy of an opportunity for education is evil.

See http://www.catwhisker.org/~david/publickey.gpg for my public key.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
URL: <http://www.baylisa.org/pipermail/baylisa/attachments/20110419/3653701f/attachment.bin>