[rvm-research] Callback: Looking for Sources of Performance Variation

> Hi Christian,
>
> There are some very well known sources of performance variation for  
> managed runtimes, such as Jikes RVM.   However, it sounds like you  
> have accounted for these.   It is a little hard to tell though.   It  
> might help if you can provide the following information:
>
> - The *exact* command line used for a specific benchmark
> - The *exact* results produced by one of your runs (ideally a log of  
> the 60 results you report below)
> - The *exact* hardware you are running on (you say it is dual core, do  
> you mean Core 2 Duo?).
>
> You should not see any significant variation if you turn the AOS  
> off.   I do this fairly routinely.   Nonetheless, I'd normally take 10  
> measurements even with the AOS off.  WIth the AOS on, I'd be inclined  
> to take 20 measurements.   I don't think you need to time the 10th  
> iteration.   Take a look at the warm-up curves in the right column of  
> this page (http://dacapo.anu.edu.au/regression/perf/2006-10-MR2.html)  
> and you'll see that steady state is reached earlier than that.   I  
> typically use the 4th iteration.
>
> Cheers,
>
> --steve
>
> On 07/05/2009, at 10:13 PM, sunai@... wrote:
>
>> This is a troubleshooting question. I am trying to run the dacapo  
>> benchmarks with an older revision (14775) of Jikes, using the  
>> ''perf'' test-run (I adapted it to run only the dacapo benchmarks),  
>> but the measurements turn out to be very unstable. E. g. for dacapo-
>> fop running the 9 warum-up + 1 timed iterations for 6 executions  
>> mostly gives me results within a limited rage (barring 3% of  
>> variation), but quite a number of measurements (about one fifth) are  
>> very far off (+>10%). I have attempted a small baseline compiler  
>> modification to safe some control flow profiling (edge counters);  
>> when I run the patched VM with this code, all measurements are  
>> catapulted into the higher ballpark.
>>
>> I have switched off the AOS recompilation, which apparently also  
>> ensures that there is no invocation threshold-based recompilation. I  
>> am running Ubuntu 7.04 in single user mode on a 2-core Intel  
>> machine. The configuration  is standard (profiled production build  
>> with classpath, dacapo 2006/10). I have not manually started  
>> additional system services (not even an Xvfb server to run dacapo-
>> chart) and, as stated above, encounter the phenomenon on the out-of-
>> the-box RVM.
>>
>> Right now, I am at a loss what might cause these fluctuations, so I  
>> am interested in any advice/ideas.
>>
>> Christian Sinschek,
>> Technische Universität Darmstadt
>> --
>> Psssst! Schon vom neuen GMX MultiMessenger gehört? Der kann`s mit  
>> allen: http://www.gmx.net/de/go/multimessenger01
>>
>> ------------------------------------------------------------------------------
>> The NEW KODAK i700 Series Scanners deliver under ANY circumstances!  
>> Your
>> production scanning environment may not be a perfect world - but  
>> thanks to
>> Kodak, there's a perfect scanner to get the job done! With the NEW  
>> KODAK i700
>> Series Scanner you'll get full speed at 300 dpi even with all image
>> processing features enabled. http://p.sf.net/sfu/kodak-com
>> _______________________________________________
>> Jikesrvm-researchers mailing list
>> Jikesrvm-researchers@...
>> https://lists.sourceforge.net/lists/listinfo/jikesrvm-researchers
>
>
> ------------------------------------------------------------------------------
> The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
> production scanning environment may not be a perfect world - but thanks to
> Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700
> Series Scanner you'll get full speed at 300 dpi even with all image
> processing features enabled. http://p.sf.net/sfu/kodak-com
> _______________________________________________
> Jikesrvm-researchers mailing list
> Jikesrvm-researchers@...
> https://lists.sourceforge.net/lists/listinfo/jikesrvm-researchers
>