Re: [rvm-research] Looking for Sources of Performance Variation

In defense of certain uses of the geo mean ...

A key property of the geo mean is that the geo mean of a
collection of ratios is equal to the geo mean of their
numerators divided by the geo mean of their denominators.
That is, the geo mean of the ratios of pairs of numbers
equals the ratio of the geo means. This suggests that
it is good for dealing with collections of ratios.

Thus:

If I run a benchmark suite of (say) 20 benchmarks, summarizing
the overall performance of the suite using the geo mean of
the individual performance times is sensible ... when desiring
to compare against runing the same suite under some different
condition (say with a new compiler optimization, or on a
different hardware platform).

If one takes the geo mean of the times under the "new" treatment
and divides that by the the geo mean of the times under the
"old" treatment, you get (I claim) a sensible summary of the
*ratios* of the performance of the benchmarks for each
treatment.

A key thing here is that one benchmark may run a lot longer than
another one -- but if I take the ratio of the geo means, that
does not matter, since what I am really computing is the the
performance different as a ratio (i.e., "new" / "old").

The geo mean also tends to prevent an outlier from dominating
the measurement of central tendency. I suppose you can like
that or dislike it, but it is true.

Beyond this, I have found the use of geo mean (on the one
hand) and arithmetic or harmonic mean (one the other hand)
to be an issue argued with religious fervor -- much passion,
not much conversion. I still stand by it as a sensible way to
summarize a benchmark suite's performance, especially for
comparing against other runs of the same suite using ratios.

Best wishes -- Eliot Moss

------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables
unlimited royalty-free distribution of the report engine
for externally facing server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
Jikesrvm-researchers mailing list
Jikesrvm-researchers@...
https://lists.sourceforge.net/lists/listinfo/jikesrvm-researchers