Opinions sought on information presentation

Wed Apr 20 11:57:40 PDT 2011

HI David,

Nice work.  I like the graph and I think it's pretty obvious to anyone
that does this kind of work what it represents.  But, I think that if
this is for non-technical audiences you might want to choose a different
style of graph.

I'm actually interested in your solution, and I have a question for you
around it.  Did you look into gathering the information via SNMP and
found that it wasn't adequate resolution for your needs or was there
some other reason for going the on-box perl/shell route?

Overall looks good tho, I can see where it would be useful for gathering
performance metrics for a workload run.

satch

On 4/19/11 9:42 PM, David Wolfskill wrote:
> While I'm not actually a sysadmin per se any more, there's a fair amount
> of overlap between what I'm doing and what I did as a sysadmin.  And
> it's possible that some of y;all might find some of it of interest,
> so....
>
> I recently was able to actually get some hardware that I'm allowed to
> reconfigure, so I've been performing series of
> as-identical-as-I-can-make-them runs of a sample workload that is of
> "significant" interest to my employer.
>
> In the process, I've been gathering information during the workload in
> question:  I cobbled up a Perl script that runs a command under the
> auspices of "/usr/bin/time -lp", parses the resulting output, then spits
> out that information, together with just about anything else I was able
> to think of that might be interesting, all in a simple "tagged" format
> that's fairly easy to parse while remaining flexible (so I can change
> the mix of stuff reported -- or the sequence of things -- with
> impunity).
>
> But before it does that, it forks a little shell script (and when the
> command under examination terminates, the Perl script kills the shell
> script).  The shell script sits mostly in the background, and
> periodically (default: every 10 seconds) grabs a handful of
> readily-available information (mostly from sysctl(8); stuff from
> "netstat -nibf inet" is included by default) and re-formats that to the
> same "tagged" format (but appends it to a different file) byeans of
> awk(1).
>
> Among the data thus retrieved are the 1-, 5-, and 15-minute load
> averages; another set that can be interesting is an ordered quintuple:
> CPU utilization counters for each of the states "system", "interrupt",
> "user", "nice", and "idle" (in that sequence).
>
> These latter are what is used to determine CPU utilization (e.g., by
> top(1) or vmstat(8)).
>
> [It turns out that I actually can (and do) also get the per-core versions
> of these counters.]
>
> In any case, in an attempt to gain a bit more understanding of the
> behavior of the system, I cobbled up a bit of code:
>
> * Some Perl, to split the multivariate reported values (such as the CPU
>   counters) into separate univariate values.
>
> * Some Perl, to read the "tagged" data & spit out a "delimited" file
>   with headers.
>
> * Some R, to accept a "data frame" (result of having read output of that
>   last) and generate a graph showing the distribution of CPU states over
>   time during the execution of the command being examined.
>
> Over the last couple of days, I hacked this last to also (optionally)
> overlay the load averages on the same graph.
>
> So I'm interested in opinions on the result.  I've placed a couple of
> lightly-sanitized graphs up at <http://www.catwhisker.org/~david/perf/>,
> and solicit comments.
>
> (I'm aware that the legends are a bit truncated.  I also gather that
> some folks may not readily distinguish between the read & the green
> colors I used: the code I wrote permits those colors to be overridden,
> but if more generally useful defaults make sense, I'd like to know.  I
> don't pretend to be an expert in data presentation.)
>
> Mostly, I'm interested in knowing if other folks perceive the way the
> information is presented as (arguably) useful and fairly clear.
>
> I haven't generated any of the per-core graphs for this exercise; if
> there's interest, I could probably be persuaded to do so.
>
> Thanks!
>
> [Oh: all of this is done in a FreeBSD environment.  Each of the tools
> used is open source.  I've indicated to my manager that I'd like to be
> able to release what I've done so others might improve it.  She hasn't
> actually raised any objection so far....]
>
> Peace,
> david