« Return to Thread: [rvm-core] New DaCapo regressions
Re: [rvm-core] New DaCapo regressions
by Steve Blackburn
::
Rate this Message:
Sigh. I did not intend to open a can of worms. Let me clarify.
What the DaCapo regressions are not:
a. they are not using Jikes RVM resources
b. they are not part of the Jikes RVM project
c. they are not intended to help debug performance regressions within
Jikes RVM
- they do not provide information on revision numbers
- they do not give performance break downs in terms of GC & mutator
- they do not give large heap, small heap regressions
- they only cover DaCapo
d. they are not a substitute for our own comprehensive regression
testing
What the DaCapo regressions are:
a. A tool for the DaCapo benchmark developers (right now, that's me)
- to characterize individual benchmarks across a spectrum of VMs
- to identify the affect of changes to the benchmarks (crucial while
developing the bms, as I am actively doing now)
- to identify regressions (ie if I've broken the suite)
b. An opportunity for VM developers to compare across VMs
c. An opportunity for VM developers to anticipate how their VM will
perform with the next DaCapo release
So yes, there are problems with our (Jikes RVM) regressions right now,
(IMHO mostly related to our reporting). I don't think this issue has
much to do with the DaCapo regressions.
As to Dave's request re geomeans, yes, I hope to have something like
that some time soon. However, in the revamp I tried hard to deliver
statistically meaningful graphs. Producing meaningful geomeans is a
little difficult since (particularly with the dacapo head) not all VMs
run all benchmarks, so you end up comparing means across different
sets, which is not particularly meaningful.
--Steve
On 05/08/2008, at 7:06 AM, David P Grove wrote:
>
> Ian wrote:
>> The new graphs are really great. I wonder if it makes sense to run
>> fewer
>> performance tests now and shift the focus of the regression machines
>> onto running sanity and stress tests. These configurations test more
>> internal assertions of the RVM and can potentially help us spot bugs
>> better. Could we use the results from DaCapo to replace the current
>> performance tests?
>
> We still need jbb and jvm98 (jvm2008...) results in addition to
> DaCapo.
> However, I agree that as the new hardware comes on line, we should
> think a
> little about how best to deploy our cycles to get best coverage.
>
> --dave
>
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's
> challenge
> Build the coolest Linux based applications with Moblin SDK & win
> great prizes
> Grand prize is a trip for two to an Open Source event anywhere in
> the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Jikesrvm-core mailing list
> Jikesrvm-core@...
> https://lists.sourceforge.net/lists/listinfo/jikesrvm-core
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Jikesrvm-core mailing list
Jikesrvm-core@...
https://lists.sourceforge.net/lists/listinfo/jikesrvm-core
What the DaCapo regressions are not:
a. they are not using Jikes RVM resources
b. they are not part of the Jikes RVM project
c. they are not intended to help debug performance regressions within
Jikes RVM
- they do not provide information on revision numbers
- they do not give performance break downs in terms of GC & mutator
- they do not give large heap, small heap regressions
- they only cover DaCapo
d. they are not a substitute for our own comprehensive regression
testing
What the DaCapo regressions are:
a. A tool for the DaCapo benchmark developers (right now, that's me)
- to characterize individual benchmarks across a spectrum of VMs
- to identify the affect of changes to the benchmarks (crucial while
developing the bms, as I am actively doing now)
- to identify regressions (ie if I've broken the suite)
b. An opportunity for VM developers to compare across VMs
c. An opportunity for VM developers to anticipate how their VM will
perform with the next DaCapo release
So yes, there are problems with our (Jikes RVM) regressions right now,
(IMHO mostly related to our reporting). I don't think this issue has
much to do with the DaCapo regressions.
As to Dave's request re geomeans, yes, I hope to have something like
that some time soon. However, in the revamp I tried hard to deliver
statistically meaningful graphs. Producing meaningful geomeans is a
little difficult since (particularly with the dacapo head) not all VMs
run all benchmarks, so you end up comparing means across different
sets, which is not particularly meaningful.
--Steve
On 05/08/2008, at 7:06 AM, David P Grove wrote:
>
> Ian wrote:
>> The new graphs are really great. I wonder if it makes sense to run
>> fewer
>> performance tests now and shift the focus of the regression machines
>> onto running sanity and stress tests. These configurations test more
>> internal assertions of the RVM and can potentially help us spot bugs
>> better. Could we use the results from DaCapo to replace the current
>> performance tests?
>
> We still need jbb and jvm98 (jvm2008...) results in addition to
> DaCapo.
> However, I agree that as the new hardware comes on line, we should
> think a
> little about how best to deploy our cycles to get best coverage.
>
> --dave
>
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's
> challenge
> Build the coolest Linux based applications with Moblin SDK & win
> great prizes
> Grand prize is a trip for two to an Open Source event anywhere in
> the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Jikesrvm-core mailing list
> Jikesrvm-core@...
> https://lists.sourceforge.net/lists/listinfo/jikesrvm-core
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Jikesrvm-core mailing list
Jikesrvm-core@...
https://lists.sourceforge.net/lists/listinfo/jikesrvm-core
« Return to Thread: [rvm-core] New DaCapo regressions
| Free embeddable forum powered by Nabble | Forum Help |
