Need reviewers - jdk testing changes 6888927

[kelly.ohair]

I need some formal reviewers on these jdk/test/Makefile changes.

More polish and changes may need to be done later, but there
is value in what I have now, and I need lots of help to improve
things (and fix some of our testcases).

Here is the bugid and webrev:

6888927: Fix jdk jtreg tests to indicate which ones need othervm, allow for use of samevm option
http://cr.openjdk.java.net/~ohair/openjdk7/jdk7-build-samevm-6888927/webrev/

The goal was to see if we could easily run most of the regression
tests in the jdk/test directory, in minimum time, and without so
much noise (e.g. do not run unstable or problematic tests).
This could then be used as a benchmark to validate some quality measure
of the jdk7 that was built.

The primary changes include:
   * New jdk/test/Makefile targets: jdk_all jdk_lang jdk_util jdk_nio ...
   * Addition of a jdk/test/ProblemList file to be used by jdk/test/Makefile

I tried to group all the tests (by directory names) into jdk_* sets, then
tried to balance them as to how long each set ran and if they could be run
with the jtreg -samevm option. The balancing was tricky, and will need more
polish, along with the virtual frame buffer idea. I also had to give up
on the awt, rmi, and swing tests until the Xvfb issues are figured out.

I chose to not actually modify the tests themselves if they needed fixing,
it was just too overwhelming. So developers and teams may want to browse the
ProblemList for their favorite tests (which aren't my favorites ;^) and
consider what they might want to do.

With this new jdk/test/Makefile, anybody can:

   cd jdk/test
   gnumake [PRODUCT_HOME=${YOUR_JDK7_HOME}] [JT_HOME=${YOUR_JTREG_HOME}] jdk_all

And run about 3,000+ tests that SHOULD PASS in roughly 2 hours if your
machine isn't too slow and old. I was able to run it in 90 minutes on a
monster OpenSolaris AMD machine which had 16Gb RAM and 16 2.2Ghz cpus.

Then I added "-j 4" to the gnumake command line and it only took 30mins!
So the various jdk_* make targets can be run in parallel.

Please let me know what you think.

-kto

[jonathan.gibbons]

Kelly,

Wow, you've been busy.

The thing that stood out to me was the use of SLASH_JAVA which is
something of a Sun legacy which doesn't apply to folk outside Sun.  And,
looking at the file as a whole, it's clear there are many different
sections, not all of which apply to the OpenJDK community.   Would it
not help to split this file up into sections for the various test
suites, or at least into a couple of files like jtreg.gmk and other.gmk
so that people outside Sun have a better chance of figuring out this file.

-- Jon


Kelly O'Hair wrote:

>
> I need some formal reviewers on these jdk/test/Makefile changes.
>
> More polish and changes may need to be done later, but there
> is value in what I have now, and I need lots of help to improve
> things (and fix some of our testcases).
>
> Here is the bugid and webrev:
>
> 6888927: Fix jdk jtreg tests to indicate which ones need othervm,
> allow for use of samevm option
> http://cr.openjdk.java.net/~ohair/openjdk7/jdk7-build-samevm-6888927/webrev/ 
>
>
> The goal was to see if we could easily run most of the regression
> tests in the jdk/test directory, in minimum time, and without so
> much noise (e.g. do not run unstable or problematic tests).
> This could then be used as a benchmark to validate some quality measure
> of the jdk7 that was built.
>
> The primary changes include:
>   * New jdk/test/Makefile targets: jdk_all jdk_lang jdk_util jdk_nio ...
>   * Addition of a jdk/test/ProblemList file to be used by
> jdk/test/Makefile
>
> I tried to group all the tests (by directory names) into jdk_* sets, then
> tried to balance them as to how long each set ran and if they could be
> run
> with the jtreg -samevm option. The balancing was tricky, and will need
> more
> polish, along with the virtual frame buffer idea. I also had to give up
> on the awt, rmi, and swing tests until the Xvfb issues are figured out.
>
> I chose to not actually modify the tests themselves if they needed
> fixing,
> it was just too overwhelming. So developers and teams may want to
> browse the
> ProblemList for their favorite tests (which aren't my favorites ;^) and
> consider what they might want to do.
>
> With this new jdk/test/Makefile, anybody can:
>
>   cd jdk/test
>   gnumake [PRODUCT_HOME=${YOUR_JDK7_HOME}]
> [JT_HOME=${YOUR_JTREG_HOME}] jdk_all
>
> And run about 3,000+ tests that SHOULD PASS in roughly 2 hours if your
> machine isn't too slow and old. I was able to run it in 90 minutes on a
> monster OpenSolaris AMD machine which had 16Gb RAM and 16 2.2Ghz cpus.
>
> Then I added "-j 4" to the gnumake command line and it only took 30mins!
> So the various jdk_* make targets can be run in parallel.
>
> Please let me know what you think.
>
> -kto
>
>

[Martin Buchholz-3]

On Tue, Nov 3, 2009 at 18:45, Jonathan Gibbons <Jonathan.Gibbons@...> wrote:

The thing that stood out to me was the use of SLASH_JAVA which is something of a Sun legacy which doesn't apply to folk outside Sun. 

I've been using SLASH_JAVA because it simplifies the job of organizing
my jdk build dependencies.  make sanity tells me where it's expecting
to find things, and I fulfill its expectations.
Admittedly, the organization is easier to understand if you've
previously worked at Sun.

Martin

[Martin Buchholz-3]

One of the things you've done is to create "test sets".
I did something like that in my "jtr" script (ask Tim if you can't find it).
For each logical component, it's non-trivial to find all the tests
for that (e.g. how to test string handling). 
This is especially true for shared directories like java/lang
and java/util.  I don't think there's much value in creating a special
target for testing java/util, since
1) jtreg can do that easily already
2) this doesn't match the job of any particular technology developer

What would be really useful is to try to fix all the tests
that fail under -samevm (perhaps by marking them
@run othervm)

Martin

On Tue, Nov 3, 2009 at 18:03, Kelly O'Hair <Kelly.Ohair@...> wrote:

I need some formal reviewers on these jdk/test/Makefile changes.

More polish and changes may need to be done later, but there
is value in what I have now, and I need lots of help to improve
things (and fix some of our testcases).

Here is the bugid and webrev:

6888927: Fix jdk jtreg tests to indicate which ones need othervm, allow for use of samevm option
http://cr.openjdk.java.net/~ohair/openjdk7/jdk7-build-samevm-6888927/webrev/

The goal was to see if we could easily run most of the regression
tests in the jdk/test directory, in minimum time, and without so
much noise (e.g. do not run unstable or problematic tests).
This could then be used as a benchmark to validate some quality measure
of the jdk7 that was built.

The primary changes include:
 * New jdk/test/Makefile targets: jdk_all jdk_lang jdk_util jdk_nio ...
 * Addition of a jdk/test/ProblemList file to be used by jdk/test/Makefile

I tried to group all the tests (by directory names) into jdk_* sets, then
tried to balance them as to how long each set ran and if they could be run
with the jtreg -samevm option. The balancing was tricky, and will need more
polish, along with the virtual frame buffer idea. I also had to give up
on the awt, rmi, and swing tests until the Xvfb issues are figured out.

I chose to not actually modify the tests themselves if they needed fixing,
it was just too overwhelming. So developers and teams may want to browse the
ProblemList for their favorite tests (which aren't my favorites ;^) and
consider what they might want to do.

With this new jdk/test/Makefile, anybody can:

 cd jdk/test
 gnumake [PRODUCT_HOME=${YOUR_JDK7_HOME}] [JT_HOME=${YOUR_JTREG_HOME}] jdk_all

And run about 3,000+ tests that SHOULD PASS in roughly 2 hours if your
machine isn't too slow and old. I was able to run it in 90 minutes on a
monster OpenSolaris AMD machine which had 16Gb RAM and 16 2.2Ghz cpus.

Then I added "-j 4" to the gnumake command line and it only took 30mins!
So the various jdk_* make targets can be run in parallel.

Please let me know what you think.

-kto

[tim.bell]

Hi Kelly
> I need some formal reviewers on these jdk/test/Makefile changes.
>
> More polish and changes may need to be done later, but there
> is value in what I have now, and I need lots of help to improve
> things (and fix some of our testcases).

This is a great start.

> Here is the bugid and webrev:
>
> 6888927: Fix jdk jtreg tests to indicate which ones need othervm, allow for use of samevm option
> http://cr.openjdk.java.net/~ohair/openjdk7/jdk7-build-samevm-6888927/webrev/
>
> The goal was to see if we could easily run most of the regression
> tests in the jdk/test directory, in minimum time, and without so
> much noise (e.g. do not run unstable or problematic tests).
> This could then be used as a benchmark to validate some quality measure
> of the jdk7 that was built.
>
> The primary changes include:
>    * New jdk/test/Makefile targets: jdk_all jdk_lang jdk_util jdk_nio ...
>    * Addition of a jdk/test/ProblemList file to be used by jdk/test/Makefile
>
> I tried to group all the tests (by directory names) into jdk_* sets, then
> tried to balance them as to how long each set ran and if they could be run
> with the jtreg -samevm option. The balancing was tricky, and will need more
> polish, along with the virtual frame buffer idea. I also had to give up
> on the awt, rmi, and swing tests until the Xvfb issues are figured out.

I wrote that start-Xvfb.sh script for you, but now I can see that I
need to take a closer look at it.  On recent OpenSolaris installs,
/usr/openwin is a symlink to /usr/X11, so my if test on line 20
is dubious at best.

I will take a look tomorrow and see if I can find a solution
that works better for Solaris 10, OpenSolaris, and Linux.

> I chose to not actually modify the tests themselves if they needed fixing,
> it was just too overwhelming. So developers and teams may want to browse the
> ProblemList for their favorite tests (which aren't my favorites ;^) and
> consider what they might want to do.
>
> With this new jdk/test/Makefile, anybody can:
>
>    cd jdk/test
>    gnumake [PRODUCT_HOME=${YOUR_JDK7_HOME}] [JT_HOME=${YOUR_JTREG_HOME}] jdk_all
>
> And run about 3,000+ tests that SHOULD PASS in roughly 2 hours if your
> machine isn't too slow and old. I was able to run it in 90 minutes on a
> monster OpenSolaris AMD machine which had 16Gb RAM and 16 2.2Ghz cpus.

For the record, that machine is a SunFire X2250 [1] which presents a total
of 16 2267 MHz CPUs to the O/S.  Kelly, Kumar, and I have been dumping work
on this system for almost a week.  It hasn't blinked - it grinds through
everything we throw at it.  I am impressed by what our Sun hardware colleagues
have been able to squeeze into one rack unit.  A few years ago, a server this
powerful would have been as large as a microwave oven, if not a refrigerator.
Thanks are due to the Sun lab managers who recently gave us two of these for
JDK7 use.

> Then I added "-j 4" to the gnumake command line and it only took 30mins!
> So the various jdk_* make targets can be run in parallel.

Very cool!  I have never attempted that before, with any of our test
suites.  In the past, too many tests left dirt behind and caused
many innocent test that might run later to fail...

Tim

[1] Community info:  http://www.sun.com/bigadmin/hcl/data/systems/details/9521.html
        Sales info: http://www.sun.com/servers/x64/x2250/

[Alan Bateman]

Kelly O'Hair wrote:

>
> I need some formal reviewers on these jdk/test/Makefile changes.
>
> More polish and changes may need to be done later, but there
> is value in what I have now, and I need lots of help to improve
> things (and fix some of our testcases).
>
> Here is the bugid and webrev:
>
> 6888927: Fix jdk jtreg tests to indicate which ones need othervm,
> allow for use of samevm option
> http://cr.openjdk.java.net/~ohair/openjdk7/jdk7-build-samevm-6888927/webrev/ 
>
>
> The goal was to see if we could easily run most of the regression
> tests in the jdk/test directory, in minimum time, and without so
> much noise (e.g. do not run unstable or problematic tests).
> This could then be used as a benchmark to validate some quality measure
> of the jdk7 that was built.
>
> The primary changes include:
>   * New jdk/test/Makefile targets: jdk_all jdk_lang jdk_util jdk_nio ...
>   * Addition of a jdk/test/ProblemList file to be used by
> jdk/test/Makefile
>
> I tried to group all the tests (by directory names) into jdk_* sets, then
> tried to balance them as to how long each set ran and if they could be
> run
> with the jtreg -samevm option. The balancing was tricky, and will need
> more
> polish, along with the virtual frame buffer idea. I also had to give up
> on the awt, rmi, and swing tests until the Xvfb issues are figured out.
>
> I chose to not actually modify the tests themselves if they needed
> fixing,
> it was just too overwhelming. So developers and teams may want to
> browse the
> ProblemList for their favorite tests (which aren't my favorites ;^) and
> consider what they might want to do.
>
> With this new jdk/test/Makefile, anybody can:
>
>   cd jdk/test
>   gnumake [PRODUCT_HOME=${YOUR_JDK7_HOME}]
> [JT_HOME=${YOUR_JTREG_HOME}] jdk_all
>
> And run about 3,000+ tests that SHOULD PASS in roughly 2 hours if your
> machine isn't too slow and old. I was able to run it in 90 minutes on a
> monster OpenSolaris AMD machine which had 16Gb RAM and 16 2.2Ghz cpus.
>
> Then I added "-j 4" to the gnumake command line and it only took 30mins!
> So the various jdk_* make targets can be run in parallel.
>
> Please let me know what you think.
>
> -kto
>

This looks really good. I don't see any obvious issues in the Makefile
and I grabbed the patch to try it and it worked great.

My only concern is that ProblemList will require ongoing maintenance. I
realize you don't want to get into changing tests and adding @ignore to
unstable tests but I think you'll need to send a broadcast mail to make
sure that folks know that their tests have problems in samevm mode, that
they need to remove from ProblemList when fixing tests, maybe they need
to add to ProblemList if changing/adding tests that take a long time, etc.

For the groupings, then the only odd one I noticed is that jdk_io
includes the javax/imagio tests. It would best to separate these as they
are very different areas.

-Alan.

[jonathan.gibbons]

Martin,

It would be wrong to *just* mark the broken tests as @run othervm, because then we would have no obvious way to distinguish between those tests which are intentionally othervm, and those tests which should be samevm but which are broken and have to be othervm "for now". 

It would be better to also mark such tests in some other way as well, perhaps using a keyword, so that such tests can be easily identified and tracked.

-- Jon

Martin Buchholz wrote:

One of the things you've done is to create "test sets".
I did something like that in my "jtr" script (ask Tim if you can't find it).
For each logical component, it's non-trivial to find all the tests
for that (e.g. how to test string handling). 
This is especially true for shared directories like java/lang
and java/util.  I don't think there's much value in creating a special
target for testing java/util, since
1) jtreg can do that easily already
2) this doesn't match the job of any particular technology developer

What would be really useful is to try to fix all the tests
that fail under -samevm (perhaps by marking them
@run othervm)

Martin

On Tue, Nov 3, 2009 at 18:03, Kelly O'Hair <Kelly.Ohair@...> wrote:

I need some formal reviewers on these jdk/test/Makefile changes.

More polish and changes may need to be done later, but there
is value in what I have now, and I need lots of help to improve
things (and fix some of our testcases).

Here is the bugid and webrev:

6888927: Fix jdk jtreg tests to indicate which ones need othervm, allow for use of samevm option
http://cr.openjdk.java.net/~ohair/openjdk7/jdk7-build-samevm-6888927/webrev/

The goal was to see if we could easily run most of the regression
tests in the jdk/test directory, in minimum time, and without so
much noise (e.g. do not run unstable or problematic tests).
This could then be used as a benchmark to validate some quality measure
of the jdk7 that was built.

The primary changes include:
 * New jdk/test/Makefile targets: jdk_all jdk_lang jdk_util jdk_nio ...
 * Addition of a jdk/test/ProblemList file to be used by jdk/test/Makefile

I tried to group all the tests (by directory names) into jdk_* sets, then
tried to balance them as to how long each set ran and if they could be run
with the jtreg -samevm option. The balancing was tricky, and will need more
polish, along with the virtual frame buffer idea. I also had to give up
on the awt, rmi, and swing tests until the Xvfb issues are figured out.

I chose to not actually modify the tests themselves if they needed fixing,
it was just too overwhelming. So developers and teams may want to browse the
ProblemList for their favorite tests (which aren't my favorites ;^) and
consider what they might want to do.

With this new jdk/test/Makefile, anybody can:

 cd jdk/test
 gnumake [PRODUCT_HOME=${YOUR_JDK7_HOME}] [JT_HOME=${YOUR_JTREG_HOME}] jdk_all

And run about 3,000+ tests that SHOULD PASS in roughly 2 hours if your
machine isn't too slow and old. I was able to run it in 90 minutes on a
monster OpenSolaris AMD machine which had 16Gb RAM and 16 2.2Ghz cpus.

Then I added "-j 4" to the gnumake command line and it only took 30mins!
So the various jdk_* make targets can be run in parallel.

Please let me know what you think.

-kto

[kelly.ohair]

Jonathan Gibbons wrote:

> Kelly,
>
> Wow, you've been busy.
>
> The thing that stood out to me was the use of SLASH_JAVA which is
> something of a Sun legacy which doesn't apply to folk outside Sun.  And,
> looking at the file as a whole, it's clear there are many different
> sections, not all of which apply to the OpenJDK community.   Would it
> not help to split this file up into sections for the various test
> suites, or at least into a couple of files like jtreg.gmk and other.gmk
> so that people outside Sun have a better chance of figuring out this file.

I took a stab at creating *.gmk files and did not like the result.
Separating this stuff into separate make include files just makes it
harder to maintain in my opinion. Completely separate Makefile's just seemed
like more work than it was worth. The sections on jck and vmsqe are isolated
as best I can in the one file, I didn't think that was so bad.

One big Makefile has some big advantages, for now, I'd like to leave it that way.

-kto

>
> -- Jon
>
>
> Kelly O'Hair wrote:
>>
>> I need some formal reviewers on these jdk/test/Makefile changes.
>>
>> More polish and changes may need to be done later, but there
>> is value in what I have now, and I need lots of help to improve
>> things (and fix some of our testcases).
>>
>> Here is the bugid and webrev:
>>
>> 6888927: Fix jdk jtreg tests to indicate which ones need othervm,
>> allow for use of samevm option
>> http://cr.openjdk.java.net/~ohair/openjdk7/jdk7-build-samevm-6888927/webrev/ 
>>
>>
>> The goal was to see if we could easily run most of the regression
>> tests in the jdk/test directory, in minimum time, and without so
>> much noise (e.g. do not run unstable or problematic tests).
>> This could then be used as a benchmark to validate some quality measure
>> of the jdk7 that was built.
>>
>> The primary changes include:
>>   * New jdk/test/Makefile targets: jdk_all jdk_lang jdk_util jdk_nio ...
>>   * Addition of a jdk/test/ProblemList file to be used by
>> jdk/test/Makefile
>>
>> I tried to group all the tests (by directory names) into jdk_* sets, then
>> tried to balance them as to how long each set ran and if they could be
>> run
>> with the jtreg -samevm option. The balancing was tricky, and will need
>> more
>> polish, along with the virtual frame buffer idea. I also had to give up
>> on the awt, rmi, and swing tests until the Xvfb issues are figured out.
>>
>> I chose to not actually modify the tests themselves if they needed
>> fixing,
>> it was just too overwhelming. So developers and teams may want to
>> browse the
>> ProblemList for their favorite tests (which aren't my favorites ;^) and
>> consider what they might want to do.
>>
>> With this new jdk/test/Makefile, anybody can:
>>
>>   cd jdk/test
>>   gnumake [PRODUCT_HOME=${YOUR_JDK7_HOME}]
>> [JT_HOME=${YOUR_JTREG_HOME}] jdk_all
>>
>> And run about 3,000+ tests that SHOULD PASS in roughly 2 hours if your
>> machine isn't too slow and old. I was able to run it in 90 minutes on a
>> monster OpenSolaris AMD machine which had 16Gb RAM and 16 2.2Ghz cpus.
>>
>> Then I added "-j 4" to the gnumake command line and it only took 30mins!
>> So the various jdk_* make targets can be run in parallel.
>>
>> Please let me know what you think.
>>
>> -kto
>>
>>
>

[jonathan.gibbons]

Kelly O'Hair wrote:

>
>
> Jonathan Gibbons wrote:
>> Kelly,
>>
>> Wow, you've been busy.
>>
>> The thing that stood out to me was the use of SLASH_JAVA which is
>> something of a Sun legacy which doesn't apply to folk outside Sun.  
>> And, looking at the file as a whole, it's clear there are many
>> different sections, not all of which apply to the OpenJDK
>> community.   Would it not help to split this file up into sections
>> for the various test suites, or at least into a couple of files like
>> jtreg.gmk and other.gmk so that people outside Sun have a better
>> chance of figuring out this file.
>
> I took a stab at creating *.gmk files and did not like the result.
> Separating this stuff into separate make include files just makes it
> harder to maintain in my opinion. Completely separate Makefile's just
> seemed
> like more work than it was worth. The sections on jck and vmsqe are
> isolated
> as best I can in the one file, I didn't think that was so bad.
>
> One big Makefile has some big advantages, for now, I'd like to leave
> it that way.
>
> -kto

OK. Thanks for trying.

-- Jon

>
>>
>> -- Jon
>>
>>
>> Kelly O'Hair wrote:
>>>
>>> I need some formal reviewers on these jdk/test/Makefile changes.
>>>
>>> More polish and changes may need to be done later, but there
>>> is value in what I have now, and I need lots of help to improve
>>> things (and fix some of our testcases).
>>>
>>> Here is the bugid and webrev:
>>>
>>> 6888927: Fix jdk jtreg tests to indicate which ones need othervm,
>>> allow for use of samevm option
>>> http://cr.openjdk.java.net/~ohair/openjdk7/jdk7-build-samevm-6888927/webrev/ 
>>>
>>>
>>> The goal was to see if we could easily run most of the regression
>>> tests in the jdk/test directory, in minimum time, and without so
>>> much noise (e.g. do not run unstable or problematic tests).
>>> This could then be used as a benchmark to validate some quality measure
>>> of the jdk7 that was built.
>>>
>>> The primary changes include:
>>>   * New jdk/test/Makefile targets: jdk_all jdk_lang jdk_util jdk_nio
>>> ...
>>>   * Addition of a jdk/test/ProblemList file to be used by
>>> jdk/test/Makefile
>>>
>>> I tried to group all the tests (by directory names) into jdk_* sets,
>>> then
>>> tried to balance them as to how long each set ran and if they could
>>> be run
>>> with the jtreg -samevm option. The balancing was tricky, and will
>>> need more
>>> polish, along with the virtual frame buffer idea. I also had to give up
>>> on the awt, rmi, and swing tests until the Xvfb issues are figured out.
>>>
>>> I chose to not actually modify the tests themselves if they needed
>>> fixing,
>>> it was just too overwhelming. So developers and teams may want to
>>> browse the
>>> ProblemList for their favorite tests (which aren't my favorites ;^) and
>>> consider what they might want to do.
>>>
>>> With this new jdk/test/Makefile, anybody can:
>>>
>>>   cd jdk/test
>>>   gnumake [PRODUCT_HOME=${YOUR_JDK7_HOME}]
>>> [JT_HOME=${YOUR_JTREG_HOME}] jdk_all
>>>
>>> And run about 3,000+ tests that SHOULD PASS in roughly 2 hours if your
>>> machine isn't too slow and old. I was able to run it in 90 minutes on a
>>> monster OpenSolaris AMD machine which had 16Gb RAM and 16 2.2Ghz cpus.
>>>
>>> Then I added "-j 4" to the gnumake command line and it only took
>>> 30mins!
>>> So the various jdk_* make targets can be run in parallel.
>>>
>>> Please let me know what you think.
>>>
>>> -kto
>>>
>>>
>>

[kelly.ohair]

Martin Buchholz wrote:
> One of the things you've done is to create "test sets".
> I did something like that in my "jtr" script (ask Tim if you can't find it).
> For each logical component, it's non-trivial to find all the tests
> for that (e.g. how to test string handling).
> This is especially true for shared directories like java/lang
> and java/util.  I don't think there's much value in creating a special
> target for testing java/util, since
> 1) jtreg can do that easily already
> 2) this doesn't match the job of any particular technology developer

With the jtreg options I set in the Makefile, there is a big
difference between just running 'jtreg java/util' vs. 'make jdk_util'.
The Makefile will exclude problem tests and @ignore tests, and
will use -samevm (which is not the jtreg default).
So I can somewhat guarantee that if all is good with the world,
'make jdk_util' will be 100% pass, and run as quickly as possible.

That's what started me on this road, running 'jtreg java/util' is
slow and unstable, and most of all, when it finishes I don't really
know if it was a good result or a bad result.

Don't mean to pick on java/util, the same is true with any of these
batches. I kept wondering, "I got N failures, is that good?" :^(

And with -samevm, it is valuable to run "test sets" as a way of
isolating the possible problems with tests sharing a VM.

>
> What would be really useful is to try to fix all the tests
> that fail under -samevm (perhaps by marking them
> @run othervm)

I started down that road myself and fell into a big rat hole. :^(
Modifying tests requires a buy in from those teams, and just the
logistics of that effort was more than I had the time for.

Also, it's not clear that just making them run othervm is the
right thing to do in some cases. In talking to some developers,
some want to go the extra mile to make the test samevm safe,
others don't. Some don't want to, or have the time, to do anything
to their tests. So it's a large mix of attitudes and plans on how
to deal with it.

So the ProblemList was my answer, and a testcase checklist for teams
that want to fix their tests, and then delete the lines from ProblemList.

I'll probably try and fix some of the tests I'm familiar with in
the future, like the serviceability tests.

-kto

>
> Martin
>
> On Tue, Nov 3, 2009 at 18:03, Kelly O'Hair <Kelly.Ohair@...
> <mailto:Kelly.Ohair@...>> wrote:
>
>
>     I need some formal reviewers on these jdk/test/Makefile changes.
>
>     More polish and changes may need to be done later, but there
>     is value in what I have now, and I need lots of help to improve
>     things (and fix some of our testcases).
>
>     Here is the bugid and webrev:
>
>     6888927: Fix jdk jtreg tests to indicate which ones need othervm,
>     allow for use of samevm option
>     http://cr.openjdk.java.net/~ohair/openjdk7/jdk7-build-samevm-6888927/webrev/
>     <http://cr.openjdk.java.net/%7Eohair/openjdk7/jdk7-build-samevm-6888927/webrev/>
>
>     The goal was to see if we could easily run most of the regression
>     tests in the jdk/test directory, in minimum time, and without so
>     much noise (e.g. do not run unstable or problematic tests).
>     This could then be used as a benchmark to validate some quality measure
>     of the jdk7 that was built.
>
>     The primary changes include:
>      * New jdk/test/Makefile targets: jdk_all jdk_lang jdk_util jdk_nio ...
>      * Addition of a jdk/test/ProblemList file to be used by
>     jdk/test/Makefile
>
>     I tried to group all the tests (by directory names) into jdk_* sets,
>     then
>     tried to balance them as to how long each set ran and if they could
>     be run
>     with the jtreg -samevm option. The balancing was tricky, and will
>     need more
>     polish, along with the virtual frame buffer idea. I also had to give up
>     on the awt, rmi, and swing tests until the Xvfb issues are figured out.
>
>     I chose to not actually modify the tests themselves if they needed
>     fixing,
>     it was just too overwhelming. So developers and teams may want to
>     browse the
>     ProblemList for their favorite tests (which aren't my favorites ;^) and
>     consider what they might want to do.
>
>     With this new jdk/test/Makefile, anybody can:
>
>      cd jdk/test
>      gnumake [PRODUCT_HOME=${YOUR_JDK7_HOME}]
>     [JT_HOME=${YOUR_JTREG_HOME}] jdk_all
>
>     And run about 3,000+ tests that SHOULD PASS in roughly 2 hours if your
>     machine isn't too slow and old. I was able to run it in 90 minutes on a
>     monster OpenSolaris AMD machine which had 16Gb RAM and 16 2.2Ghz cpus.
>
>     Then I added "-j 4" to the gnumake command line and it only took 30mins!
>     So the various jdk_* make targets can be run in parallel.
>
>     Please let me know what you think.
>
>     -kto
>
>
>

[kelly.ohair]

Tim Bell wrote:

> Hi Kelly
>> I need some formal reviewers on these jdk/test/Makefile changes.
>>
>> More polish and changes may need to be done later, but there
>> is value in what I have now, and I need lots of help to improve
>> things (and fix some of our testcases).
>
> This is a great start.
>
>> Here is the bugid and webrev:
>>
>> 6888927: Fix jdk jtreg tests to indicate which ones need othervm, allow for use of samevm option
>> http://cr.openjdk.java.net/~ohair/openjdk7/jdk7-build-samevm-6888927/webrev/
>>
>> The goal was to see if we could easily run most of the regression
>> tests in the jdk/test directory, in minimum time, and without so
>> much noise (e.g. do not run unstable or problematic tests).
>> This could then be used as a benchmark to validate some quality measure
>> of the jdk7 that was built.
>>
>> The primary changes include:
>>    * New jdk/test/Makefile targets: jdk_all jdk_lang jdk_util jdk_nio ...
>>    * Addition of a jdk/test/ProblemList file to be used by jdk/test/Makefile
>>
>> I tried to group all the tests (by directory names) into jdk_* sets, then
>> tried to balance them as to how long each set ran and if they could be run
>> with the jtreg -samevm option. The balancing was tricky, and will need more
>> polish, along with the virtual frame buffer idea. I also had to give up
>> on the awt, rmi, and swing tests until the Xvfb issues are figured out.
>
> I wrote that start-Xvfb.sh script for you, but now I can see that I
> need to take a closer look at it.  On recent OpenSolaris installs,
> /usr/openwin is a symlink to /usr/X11, so my if test on line 20
> is dubious at best.

I had to tweek that part of it, I wanted to favor /usr/X11, then go
after /usr/openwin.

>
> I will take a look tomorrow and see if I can find a solution
> that works better for Solaris 10, OpenSolaris, and Linux.

I got it working, I think, it was a little tricky to get it to die,
probably need a trap line in the script to make sure that when the
script dies, it takes the display and manager with it.

But the bigger problem was that the awt tests and even swing tests
seem to be able to bring the Xvfb down easily.

>
>
>> I chose to not actually modify the tests themselves if they needed fixing,
>> it was just too overwhelming. So developers and teams may want to browse the
>> ProblemList for their favorite tests (which aren't my favorites ;^) and
>> consider what they might want to do.
>>
>> With this new jdk/test/Makefile, anybody can:
>>
>>    cd jdk/test
>>    gnumake [PRODUCT_HOME=${YOUR_JDK7_HOME}] [JT_HOME=${YOUR_JTREG_HOME}] jdk_all
>>
>> And run about 3,000+ tests that SHOULD PASS in roughly 2 hours if your
>> machine isn't too slow and old. I was able to run it in 90 minutes on a
>> monster OpenSolaris AMD machine which had 16Gb RAM and 16 2.2Ghz cpus.
>
> For the record, that machine is a SunFire X2250 [1] which presents a total
> of 16 2267 MHz CPUs to the O/S.  Kelly, Kumar, and I have been dumping work
> on this system for almost a week.  It hasn't blinked - it grinds through
> everything we throw at it.  I am impressed by what our Sun hardware colleagues
> have been able to squeeze into one rack unit.  A few years ago, a server this
> powerful would have been as large as a microwave oven, if not a refrigerator.
> Thanks are due to the Sun lab managers who recently gave us two of these for
> JDK7 use.
>
>> Then I added "-j 4" to the gnumake command line and it only took 30mins!
>> So the various jdk_* make targets can be run in parallel.
>
> Very cool!  I have never attempted that before, with any of our test
> suites.  In the past, too many tests left dirt behind and caused
> many innocent test that might run later to fail...

Getting 3,000+ stable tests run in 30mins will be handy.

Managing the ProblemList (as Jon noted) will be a pain at times.

-kto

>
> Tim
>
> [1] Community info:  http://www.sun.com/bigadmin/hcl/data/systems/details/9521.html
>         Sales info: http://www.sun.com/servers/x64/x2250/

[kelly.ohair]

Alan Bateman wrote:

> Kelly O'Hair wrote:
>>
>> I need some formal reviewers on these jdk/test/Makefile changes.
>>
>> More polish and changes may need to be done later, but there
>> is value in what I have now, and I need lots of help to improve
>> things (and fix some of our testcases).
>>
>> Here is the bugid and webrev:
>>
>> 6888927: Fix jdk jtreg tests to indicate which ones need othervm,
>> allow for use of samevm option
>> http://cr.openjdk.java.net/~ohair/openjdk7/jdk7-build-samevm-6888927/webrev/ 
>>
>>
>> The goal was to see if we could easily run most of the regression
>> tests in the jdk/test directory, in minimum time, and without so
>> much noise (e.g. do not run unstable or problematic tests).
>> This could then be used as a benchmark to validate some quality measure
>> of the jdk7 that was built.
>>
>> The primary changes include:
>>   * New jdk/test/Makefile targets: jdk_all jdk_lang jdk_util jdk_nio ...
>>   * Addition of a jdk/test/ProblemList file to be used by
>> jdk/test/Makefile
>>
>> I tried to group all the tests (by directory names) into jdk_* sets, then
>> tried to balance them as to how long each set ran and if they could be
>> run
>> with the jtreg -samevm option. The balancing was tricky, and will need
>> more
>> polish, along with the virtual frame buffer idea. I also had to give up
>> on the awt, rmi, and swing tests until the Xvfb issues are figured out.
>>
>> I chose to not actually modify the tests themselves if they needed
>> fixing,
>> it was just too overwhelming. So developers and teams may want to
>> browse the
>> ProblemList for their favorite tests (which aren't my favorites ;^) and
>> consider what they might want to do.
>>
>> With this new jdk/test/Makefile, anybody can:
>>
>>   cd jdk/test
>>   gnumake [PRODUCT_HOME=${YOUR_JDK7_HOME}]
>> [JT_HOME=${YOUR_JTREG_HOME}] jdk_all
>>
>> And run about 3,000+ tests that SHOULD PASS in roughly 2 hours if your
>> machine isn't too slow and old. I was able to run it in 90 minutes on a
>> monster OpenSolaris AMD machine which had 16Gb RAM and 16 2.2Ghz cpus.
>>
>> Then I added "-j 4" to the gnumake command line and it only took 30mins!
>> So the various jdk_* make targets can be run in parallel.
>>
>> Please let me know what you think.
>>
>> -kto
>>
> This looks really good. I don't see any obvious issues in the Makefile
> and I grabbed the patch to try it and it worked great.
>
> My only concern is that ProblemList will require ongoing maintenance. I
> realize you don't want to get into changing tests and adding @ignore to
> unstable tests but I think you'll need to send a broadcast mail to make
> sure that folks know that their tests have problems in samevm mode, that
> they need to remove from ProblemList when fixing tests, maybe they need
> to add to ProblemList if changing/adding tests that take a long time, etc.

Yeah, but I didn't feel like I had many good solutions to this issue.

I'll maintain it to some degree, but I need the people that use this
makefile to help me out, and for everyone to start looking at why
their tests are even on this list.

>
> For the groupings, then the only odd one I noticed is that jdk_io
> includes the javax/imagio tests. It would best to separate these as they
> are very different areas.

It was more of a balancing act on the test groupings with regards to timings
etc., and imageio did have "io" in it's name :^), and most of them ran safely
in samevm mode. So we can move them to any other group that is run in samevm
mode.  Do you have a better match than jdk_io?


-kto

>
> -Alan.

[gnu_andrew]

2009/11/4 Kelly O'Hair <Kelly.Ohair@...>:

>
> I need some formal reviewers on these jdk/test/Makefile changes.
>
> More polish and changes may need to be done later, but there
> is value in what I have now, and I need lots of help to improve
> things (and fix some of our testcases).
>
> Here is the bugid and webrev:
>
> 6888927: Fix jdk jtreg tests to indicate which ones need othervm, allow for
> use of samevm option
> http://cr.openjdk.java.net/~ohair/openjdk7/jdk7-build-samevm-6888927/webrev/
>
> The goal was to see if we could easily run most of the regression
> tests in the jdk/test directory, in minimum time, and without so
> much noise (e.g. do not run unstable or problematic tests).
> This could then be used as a benchmark to validate some quality measure
> of the jdk7 that was built.
>
> The primary changes include:
>  * New jdk/test/Makefile targets: jdk_all jdk_lang jdk_util jdk_nio ...
>  * Addition of a jdk/test/ProblemList file to be used by jdk/test/Makefile
>
> I tried to group all the tests (by directory names) into jdk_* sets, then
> tried to balance them as to how long each set ran and if they could be run
> with the jtreg -samevm option. The balancing was tricky, and will need more
> polish, along with the virtual frame buffer idea. I also had to give up
> on the awt, rmi, and swing tests until the Xvfb issues are figured out.
>
> I chose to not actually modify the tests themselves if they needed fixing,
> it was just too overwhelming. So developers and teams may want to browse the
> ProblemList for their favorite tests (which aren't my favorites ;^) and
> consider what they might want to do.
>
> With this new jdk/test/Makefile, anybody can:
>
>  cd jdk/test
>  gnumake [PRODUCT_HOME=${YOUR_JDK7_HOME}] [JT_HOME=${YOUR_JTREG_HOME}]
> jdk_all
>
> And run about 3,000+ tests that SHOULD PASS in roughly 2 hours if your
> machine isn't too slow and old. I was able to run it in 90 minutes on a
> monster OpenSolaris AMD machine which had 16Gb RAM and 16 2.2Ghz cpus.
>
> Then I added "-j 4" to the gnumake command line and it only took 30mins!
> So the various jdk_* make targets can be run in parallel.
>
> Please let me know what you think.
>
> -kto
>
>
>

I like this idea, the time to run a jdk jtreg test has put me off in the past.
BTW, jtreg is the one thing I wouldn't mind having in the tree as it's
not something people are likely to have installed.  IcedTea has a cut
down copy and that means tests get run.

I'm worried about the use of the IcedTea7 test list.  These were added
just because they were the ones that failed under my testing and
that's a while ago.  Just as may happen with your ProblemList, this
hasn't been maintained recently so I'd be wary about including the
list verbatim.

With this patch in, IcedTea7 can just use the list provided here.  We
don't use test/Makefile at present at all, and it would be nice to do
so.  I need to look at the issues with that.
--
Andrew :-)

Free Java Software Engineer
Red Hat, Inc. (http://www.redhat.com)

Support Free Java!
Contribute to GNU Classpath and the OpenJDK
http://www.gnu.org/software/classpath
http://openjdk.java.net

PGP Key: 94EFD9D8 (http://subkeys.pgp.net)
Fingerprint: F8EF F1EA 401E 2E60 15FA  7927 142C 2591 94EF D9D8

[Alan Bateman]

Kelly O'Hair wrote:
> :
> It was more of a balancing act on the test groupings with regards to
> timings
> etc., and imageio did have "io" in it's name :^), and most of them ran
> safely
> in samevm mode. So we can move them to any other group that is run in
> samevm
> mode.  Do you have a better match than jdk_io?
It's probably best to group the image I/O tests with other tests in the
client area. So far I see you've got jdk_awt and jdk_swing. Would the
timing lean themselves to adding a new target or adding to one of the
existing targets? If reducing the test list for jdk_io causes some
unbalance then you could add to it the test list from jdk_nio1 (the
java/nio/file tests) - they are in good shape and run quickly.

-Alan.

[kelly.ohair]

Alan Bateman wrote:

> Kelly O'Hair wrote:
>> :
>> It was more of a balancing act on the test groupings with regards to
>> timings
>> etc., and imageio did have "io" in it's name :^), and most of them ran
>> safely
>> in samevm mode. So we can move them to any other group that is run in
>> samevm
>> mode.  Do you have a better match than jdk_io?
> It's probably best to group the image I/O tests with other tests in the
> client area. So far I see you've got jdk_awt and jdk_swing. Would the
> timing lean themselves to adding a new target or adding to one of the
> existing targets? If reducing the test list for jdk_io causes some
> unbalance then you could add to it the test list from jdk_nio1 (the
> java/nio/file tests) - they are in good shape and run quickly.
>

I'll move them to jdk_misc for now. Only 22 tests total anyway.

-kto

> -Alan.
>
>

[Martin Buchholz-3]

On Wed, Nov 4, 2009 at 09:19, Kelly O'Hair <Kelly.Ohair@...> wrote:

> Managing the ProblemList (as Jon noted) will be a pain at times.

It's even more of a problem as development becomes more distributed,
as failures become platform-dependent, etc...

I would prefer to see per-file annotations that jtreg can understand.

I also like the approach I advocated some years ago:
- run tests under JDK 1
- run tests under JDK 2
- diff the results (JTreport directories)
See my diff-javatest script.
jtreg now has support for this (I haven't tried it lately).

Martin

[kelly.ohair]

Martin Buchholz wrote:
> On Wed, Nov 4, 2009 at 09:19, Kelly O'Hair <Kelly.Ohair@...> wrote:
>
>> Managing the ProblemList (as Jon noted) will be a pain at times.
>
> It's even more of a problem as development becomes more distributed,
> as failures become platform-dependent, etc...
>
> I would prefer to see per-file annotations that jtreg can understand.

Not sure how annotating tests is any better in a distributed world,
or if tests failed in a platform-dependent way. Both seem nasty.

I have no objection and would actually prefer to see the testcases
annotated, with @ignore or whatever keywords people want to invent.
But I came to the conclusion that in general, relying on that as the sole
way to exclude unstable tests was just not good enough, and may not happen
for a long long time.
I would hope that eventually, this ProblemList shrinks, perhaps to
nothingness someday, I know I know,  ... dream on. :^(

The issue I saw was that regressions could sneak in all kinds of ways,
and at any given point in time, a testcase could become unstable
due to an OS patch, or a hotspot change, or something outside of the
jdk repository.

So we need a quick and easy way to exclude, and stop having dozens
of developers investigate the same known failures.
Yes, in many cases bugs should be filed, jdk bugs or testcase bugs,
but my first priority was to find that list of tests that pass reliably,
and stick my wizard's staff in the ground and say 'Thou shall pass!'. ;^)

Then see if we could start requiring a set of tests to always pass.

Next, I start going through the ProblemList in detail, with various
team help if possible, and see if we can shrink this problem down.

>
> I also like the approach I advocated some years ago:
> - run tests under JDK 1
> - run tests under JDK 2
> - diff the results (JTreport directories)
> See my diff-javatest script.
> jtreg now has support for this (I haven't tried it lately).

So you want everyone to run the tests twice?
Why run 6,000+ tests when you could run 3,000+ tests?

Now running all the ProblemList tests this way might be interesting,
if some of them didn't crash jtreg. :^(

-kto

>
> Martin

[jonathan.gibbons]

Kelly O'Hair wrote:
>
>
>>
>> I also like the approach I advocated some years ago:
>> - run tests under JDK 1
>> - run tests under JDK 2
>> - diff the results (JTreport directories)
>> See my diff-javatest script.
>> jtreg now has support for this (I haven't tried it lately).

In an ideal world, one set of results would be the set of results posted
publicly somewhere for the latest promoted build.  But we don't seem to
be able to make that happen, so I agree with Kelly that for now we need
a more pragmatic approach of defining the set of tests we expect to work
on every platform on every build.  Hopefully, over time, the relevant
engineers will aspire to have their tests included in this set of tests.


-- Jon

[Martin Buchholz-3]

On Wed, Nov 4, 2009 at 16:49, Jonathan Gibbons <Jonathan.Gibbons@...> wrote:
> Kelly O'Hair wrote:
> In an ideal world, one set of results would be the set of results posted
> publicly somewhere for the latest promoted build.  But we don't seem to be

I agree with the goals, and your assessment of the reality.

> able to make that happen, so I agree with Kelly that for now we need a more
> pragmatic approach of defining the set of tests we expect to work on every
> platform on every build.  Hopefully, over time, the relevant engineers will
> aspire to have their tests included in this set of tests.

I would use the ProblemList if there was an easy way to use it to filter
my jtreg command

The most recent jtreg command I ran looked like this:

jtreg -v:nopass,fail -vmoption:-enablesystemassertions -automatic
"-k:\!ignore" test/java/nio test/sun/nio test/java/util/EnumSet
test/java/util/Vector test/java/util/HashMap
test/java/util/Collections test/java/util/WeakHashMap
test/java/util/Deque test/java/util/AbstractSequentialList
test/java/util/LinkedHashMap test/java/util/BitSet
test/java/util/AbstractList test/java/util/NavigableMap
test/java/util/AbstractCollection test/java/util/Arrays
test/java/util/LinkedList test/java/util/ArrayList
test/java/util/TreeMap test/java/util/Collection
test/java/util/AbstractMap test/java/util/LinkedHashSet
test/java/util/Hashtable test/java/util/IdentityHashMap
test/java/util/PriorityQueue test/java/util/EnumMap
test/java/util/List test/java/util/Map test/java/util/concurrent

Most of my failures were due to inability to contact javaweb.sfbay
which is probably not being addressed by ProblemList.

(BTW I am not seriously objecting to this change,
just throwing in my own personal experience)

Martin

[Joe Darcy]

Jonathan Gibbons wrote:

> Kelly O'Hair wrote:
>>
>>
>>>
>>> I also like the approach I advocated some years ago:
>>> - run tests under JDK 1
>>> - run tests under JDK 2
>>> - diff the results (JTreport directories)
>>> See my diff-javatest script.
>>> jtreg now has support for this (I haven't tried it lately).
>
> In an ideal world, one set of results would be the set of results
> posted publicly somewhere for the latest promoted build.  

Yes, I do this for the OpenJDK 6 builds, e.g.

http://blogs.sun.com/darcy/entry/openjdk_6_b17_regression_tests

and I think analogous information about JDK 7 builds would be a fine
addition.

-Joe