Nabble - OpenJDK jtreg Test Harness

Re: jtreg throws StackOverflowError when writing xml report

2009-02-19T14:00:12Z

Hi Jon,

Jonathan Gibbons wrote:
> Thanks for the update; I was expecting something around
> java.lang.Character myself,
> but definitely that sort of problem. Do you want to file a bug against
> the XML encoder?

I have filed a bug and proposed a patch at
http://icedtea.classpath.org/bugzilla/show_bug.cgi?id=295. This fixes the
test/java/beans/PropertyEditor/TestStringClassJava.java issue, but I
haven't tested it thoroughly.

Cheers,
Omair

Re: jtreg throws StackOverflowError when writing xml report

2009-02-15T16:49:39Z

Martin,

Given the [SNIP] in the middle of the trace, I would think we are seeing
>> com.sun.org.apache.xml.internal.serializer.EncodingInfo
>> $EncodingImpl.isInEncoding(EncodingInfo.java:181)
invoking itself recursively.

-- Jon

On Feb 15, 2009, at 4:46 PM, Martin Buchholz wrote:

> There have been recent changes to the
> UTF_8 encoder in OpenJDK that might be responsible.
> You can try to isolate the exact change to UTF_8 that
> could cause this difference, and then create a proper test case.
>
> changeset: 497:3dcc69147ff9
> user: sherman
> date: Fri Aug 22 14:37:46 2008 -0700
> summary: 4486841: UTF-8 decoder should adhere to corrigendum to
> Unicode 3.0.1
>
> It is surprising that one can get StackOverflowError without
> a very long stack trace of methods invoking themselves recursively,
> which is not what we're seeing.
>
> Martin
>
> On Wed, Feb 11, 2009 at 10:38, Omair Majid <omajid@...> wrote:
>> Hi all,
>>
>> I am trying to generate xml reports from jtreg. The problem is that
>> jtreg
>> throws a StackOverflowError when writing the output xml file when
>> running
>> the jdk tests. The xml report works fine for the 16 compiler tests
>> and 1351
>> langtools tests.
>>
>> I am using Icedtea6 to run the tests. These are the commands it
>> executes:
>>
>> $ mkdir -p test/jdk/JTwork test/jdk/JTreport
>> $ /notnfs/omajid/icedtea6/bootstrap/jdk1.6.0/bin/java \
>> -Djavatest.report.kinds="xml text html" \
>> -jar test/jtreg.jar -v1 -a -ignore:quiet \
>> -w:test/jdk/JTwork -r:test/jdk/JTreport \
>> -s -jdk:`pwd`/openjdk/control/build/linux-i586/j2sdk-image \
>> `pwd`/openjdk/jdk/test \
>> | tee test/check-jdk.log
>>
>> The error appears after the jtreg run completes:
>>
>> [ lots of output ]
>> Passed: vm/verifier/VerifyProtectedConstructor.java
>> Passed: vm/verifier/VerifyStackForExceptionHandlers.java
>> Test results: passed: 3,306; failed: 73; error: 5
>> Exception in thread "main" java.lang.StackOverflowError
>> at sun.nio.cs.UTF_8.updatePositions(UTF_8.java:76)
>> at sun.nio.cs.UTF_8$Encoder.encodeArrayLoop(UTF_8.java:411)
>> at sun.nio.cs.UTF_8$Encoder.encodeLoop(UTF_8.java:466)
>> at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:561)
>> at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:
>> 258)
>> at java.lang.StringCoding.encode(StringCoding.java:290)
>> at java.lang.String.getBytes(String.java:954)
>> at
>> com
>> .sun
>> .org
>> .apache
>> .xml.internal.serializer.EncodingInfo.inEncoding(EncodingInfo.java:
>> 413)
>> at
>> com.sun.org.apache.xml.internal.serializer.EncodingInfo.access
>> $100(EncodingInfo.java:62)
>> at
>> com.sun.org.apache.xml.internal.serializer.EncodingInfo
>> $EncodingImpl.isInEncoding(EncodingInfo.java:205)
>> at
>> com.sun.org.apache.xml.internal.serializer.EncodingInfo
>> $EncodingImpl.isInEncoding(EncodingInfo.java:181)
>> at
>> com.sun.org.apache.xml.internal.serializer.EncodingInfo
>> $EncodingImpl.isInEncoding(EncodingInfo.java:181)
>> [...SNIP...]
>> at
>> com.sun.org.apache.xml.internal.serializer.EncodingInfo
>> $EncodingImpl.isInEncoding(EncodingInfo.java:181)
>>
>>
>> Has anyone ever seen this before?
>>
>> Thanks,
>> Omair
>>
>>

Re: jtreg throws StackOverflowError when writing xml report

2009-02-15T16:46:35Z

There have been recent changes to the
UTF_8 encoder in OpenJDK that might be responsible.
You can try to isolate the exact change to UTF_8 that
could cause this difference, and then create a proper test case.

changeset: 497:3dcc69147ff9
user: sherman
date: Fri Aug 22 14:37:46 2008 -0700
summary: 4486841: UTF-8 decoder should adhere to corrigendum to
Unicode 3.0.1

It is surprising that one can get StackOverflowError without
a very long stack trace of methods invoking themselves recursively,
which is not what we're seeing.

Martin

On Wed, Feb 11, 2009 at 10:38, Omair Majid <omajid@...> wrote:

> Hi all,
>
> I am trying to generate xml reports from jtreg. The problem is that jtreg
> throws a StackOverflowError when writing the output xml file when running
> the jdk tests. The xml report works fine for the 16 compiler tests and 1351
> langtools tests.
>
> I am using Icedtea6 to run the tests. These are the commands it executes:
>
> $ mkdir -p test/jdk/JTwork test/jdk/JTreport
> $ /notnfs/omajid/icedtea6/bootstrap/jdk1.6.0/bin/java \
> -Djavatest.report.kinds="xml text html" \
> -jar test/jtreg.jar -v1 -a -ignore:quiet \
> -w:test/jdk/JTwork -r:test/jdk/JTreport \
> -s -jdk:`pwd`/openjdk/control/build/linux-i586/j2sdk-image \
> `pwd`/openjdk/jdk/test \
> | tee test/check-jdk.log
>
> The error appears after the jtreg run completes:
>
> [ lots of output ]
> Passed: vm/verifier/VerifyProtectedConstructor.java
> Passed: vm/verifier/VerifyStackForExceptionHandlers.java
> Test results: passed: 3,306; failed: 73; error: 5
> Exception in thread "main" java.lang.StackOverflowError
> at sun.nio.cs.UTF_8.updatePositions(UTF_8.java:76)
> at sun.nio.cs.UTF_8$Encoder.encodeArrayLoop(UTF_8.java:411)
> at sun.nio.cs.UTF_8$Encoder.encodeLoop(UTF_8.java:466)
> at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:561)
> at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:258)
> at java.lang.StringCoding.encode(StringCoding.java:290)
> at java.lang.String.getBytes(String.java:954)
> at
> com.sun.org.apache.xml.internal.serializer.EncodingInfo.inEncoding(EncodingInfo.java:413)
> at
> com.sun.org.apache.xml.internal.serializer.EncodingInfo.access$100(EncodingInfo.java:62)
> at
> com.sun.org.apache.xml.internal.serializer.EncodingInfo$EncodingImpl.isInEncoding(EncodingInfo.java:205)
> at
> com.sun.org.apache.xml.internal.serializer.EncodingInfo$EncodingImpl.isInEncoding(EncodingInfo.java:181)
> at
> com.sun.org.apache.xml.internal.serializer.EncodingInfo$EncodingImpl.isInEncoding(EncodingInfo.java:181)
> [...SNIP...]
> at
> com.sun.org.apache.xml.internal.serializer.EncodingInfo$EncodingImpl.isInEncoding(EncodingInfo.java:181)
>
>
> Has anyone ever seen this before?
>
> Thanks,
> Omair
>
>

Re: jtreg throws StackOverflowError when writing xml report

2009-02-12T11:36:11Z

Thanks for the update; I was expecting something around
java.lang.Character myself,
but definitely that sort of problem. Do you want to file a bug against
the XML encoder?

-- Jon

Omair Majid wrote:

> Hi Jon,
>
> Jonathan Gibbons wrote:
>> I suspect there is some specific test or tests in the test suite
>> causing output that it provoking this error in the xml encoder.
>> I doubt that it is the quantity of tests in the entire jdk suite that
>> is causing the issue.
>>
>> Would it be possible for you to try and narrow it down by running
>> jtreg on (just) each of the top level subdirectories of test/
>> and then if one shows the fault, on the each of the subdirectories of
>> that, and so on?
>
> Thanks for that bit of advice. The problem seems to be in
> test/java/beans/PropertyEditor/TestStringClassJava.java. Mark Wielaard
> pointed out that the test generates a string with characters 0x1000 to
> 0x1 and prints it to stdout. It appears the xml parser trips over
> that. Removing the output fixes the problem.
>
> Cheers,
> Omair

Re: jtreg throws StackOverflowError when writing xml report

2009-02-12T10:04:31Z

Hi Jon,

Jonathan Gibbons wrote:
> I suspect there is some specific test or tests in the test suite causing
> output that it provoking this error in the xml encoder.
> I doubt that it is the quantity of tests in the entire jdk suite that is
> causing the issue.
>
> Would it be possible for you to try and narrow it down by running jtreg
> on (just) each of the top level subdirectories of test/
> and then if one shows the fault, on the each of the subdirectories of
> that, and so on?

Thanks for that bit of advice. The problem seems to be in
test/java/beans/PropertyEditor/TestStringClassJava.java. Mark Wielaard
pointed out that the test generates a string with characters 0x1000 to
0x1 and prints it to stdout. It appears the xml parser trips over that.
Removing the output fixes the problem.

Cheers,
Omair

Re: jtreg throws StackOverflowError when writing xml report

2009-02-12T07:54:07Z

Omair,

I suspect there is some specific test or tests in the test suite
causing output that it provoking this error in the xml encoder.
I doubt that it is the quantity of tests in the entire jdk suite that
is causing the issue.

Would it be possible for you to try and narrow it down by running
jtreg on (just) each of the top level subdirectories of test/
and then if one shows the fault, on the each of the subdirectories of
that, and so on?

-- Jon

On Feb 12, 2009, at 7:33 AM, Omair Majid wrote:

> Hi Jon,
>
> Jonathan Gibbons wrote:
>> Since you're using IcedTea to run jtreg, I would suggest you
>> investigate the
>> possibility that this is an IcedTea issue.
>> The bug would appear to be in
>> com.sun.org.apache.xml.internal.serializer.EncodingInfo
>> $EncodingImpl.isInEncoding so it does not look like a jtreg or
>> jtharness issue.
>> Can I suggest you try and use Sun's JDK to run jtreg, while still
>> using the
>> JDK of your choice to execute the tests.
>
> I was wondering if you have run the entire jdk suite using jtreg
> with the -Djavatest.report.kinds="xml text html" option. It is only
> when using this option that the jtreg crashes.
>
> Thanks,
> Omair

Re: jtreg throws StackOverflowError when writing xml report

2009-02-12T07:33:03Z

Hi Jon,

Jonathan Gibbons wrote:

> Since you're using IcedTea to run jtreg, I would suggest you investigate
> the
> possibility that this is an IcedTea issue.
>
> The bug would appear to be in
> com.sun.org.apache.xml.internal.serializer.EncodingInfo$EncodingImpl.isInEncoding
>
> so it does not look like a jtreg or jtharness issue.
>
> Can I suggest you try and use Sun's JDK to run jtreg, while still using the
> JDK of your choice to execute the tests.

I was wondering if you have run the entire jdk suite using jtreg with
the -Djavatest.report.kinds="xml text html" option. It is only when
using this option that the jtreg crashes.

Thanks,
Omair

Re: jtreg throws StackOverflowError when writing xml report

2009-02-12T07:12:49Z

Hi Jon,

Jonathan Gibbons wrote:

I get the exact same behaviour when using Sun's JDK (build 1.6.0_12-b04)
to run jtreg.

Omair

Re: jtreg throws StackOverflowError when writing xml report

2009-02-11T10:57:51Z

Omair,

Since you're using IcedTea to run jtreg, I would suggest you investigate the
possibility that this is an IcedTea issue.

The bug would appear to be in
com.sun.org.apache.xml.internal.serializer.EncodingInfo$EncodingImpl.isInEncoding
so it does not look like a jtreg or jtharness issue.

Can I suggest you try and use Sun's JDK to run jtreg, while still using the
JDK of your choice to execute the tests.

-- Jon

Omair Majid wrote:

> Hi all,
>
> I am trying to generate xml reports from jtreg. The problem is that
> jtreg throws a StackOverflowError when writing the output xml file
> when running the jdk tests. The xml report works fine for the 16
> compiler tests and 1351 langtools tests.
>
> I am using Icedtea6 to run the tests. These are the commands it executes:
>
> $ mkdir -p test/jdk/JTwork test/jdk/JTreport
> $ /notnfs/omajid/icedtea6/bootstrap/jdk1.6.0/bin/java \
> -Djavatest.report.kinds="xml text html" \
> -jar test/jtreg.jar -v1 -a -ignore:quiet \
> -w:test/jdk/JTwork -r:test/jdk/JTreport \
> -s -jdk:`pwd`/openjdk/control/build/linux-i586/j2sdk-image \
> `pwd`/openjdk/jdk/test \
> | tee test/check-jdk.log
>
> The error appears after the jtreg run completes:
>
> [ lots of output ]
> Passed: vm/verifier/VerifyProtectedConstructor.java
> Passed: vm/verifier/VerifyStackForExceptionHandlers.java
> Test results: passed: 3,306; failed: 73; error: 5
> Exception in thread "main" java.lang.StackOverflowError
> at sun.nio.cs.UTF_8.updatePositions(UTF_8.java:76)
> at sun.nio.cs.UTF_8$Encoder.encodeArrayLoop(UTF_8.java:411)
> at sun.nio.cs.UTF_8$Encoder.encodeLoop(UTF_8.java:466)
> at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:561)
> at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:258)
> at java.lang.StringCoding.encode(StringCoding.java:290)
> at java.lang.String.getBytes(String.java:954)
> at
> com.sun.org.apache.xml.internal.serializer.EncodingInfo.inEncoding(EncodingInfo.java:413)
>
> at
> com.sun.org.apache.xml.internal.serializer.EncodingInfo.access$100(EncodingInfo.java:62)
>
> at
> com.sun.org.apache.xml.internal.serializer.EncodingInfo$EncodingImpl.isInEncoding(EncodingInfo.java:205)
>
> at
> com.sun.org.apache.xml.internal.serializer.EncodingInfo$EncodingImpl.isInEncoding(EncodingInfo.java:181)
>
> at
> com.sun.org.apache.xml.internal.serializer.EncodingInfo$EncodingImpl.isInEncoding(EncodingInfo.java:181)
>
> [...SNIP...]
> at
> com.sun.org.apache.xml.internal.serializer.EncodingInfo$EncodingImpl.isInEncoding(EncodingInfo.java:181)
>
>
>
> Has anyone ever seen this before?
>
> Thanks,
> Omair
>

jtreg throws StackOverflowError when writing xml report

2009-02-11T10:38:17Z

Hi all,

I am trying to generate xml reports from jtreg. The problem is that
jtreg throws a StackOverflowError when writing the output xml file when
running the jdk tests. The xml report works fine for the 16 compiler
tests and 1351 langtools tests.

I am using Icedtea6 to run the tests. These are the commands it executes:

$ mkdir -p test/jdk/JTwork test/jdk/JTreport
$ /notnfs/omajid/icedtea6/bootstrap/jdk1.6.0/bin/java \
-Djavatest.report.kinds="xml text html" \
-jar test/jtreg.jar -v1 -a -ignore:quiet \
-w:test/jdk/JTwork -r:test/jdk/JTreport \
-s -jdk:`pwd`/openjdk/control/build/linux-i586/j2sdk-image \
`pwd`/openjdk/jdk/test \
| tee test/check-jdk.log

The error appears after the jtreg run completes:

[ lots of output ]
Passed: vm/verifier/VerifyProtectedConstructor.java
Passed: vm/verifier/VerifyStackForExceptionHandlers.java
Test results: passed: 3,306; failed: 73; error: 5
Exception in thread "main" java.lang.StackOverflowError
at sun.nio.cs.UTF_8.updatePositions(UTF_8.java:76)
at sun.nio.cs.UTF_8$Encoder.encodeArrayLoop(UTF_8.java:411)
at sun.nio.cs.UTF_8$Encoder.encodeLoop(UTF_8.java:466)
at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:561)
at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:258)
at java.lang.StringCoding.encode(StringCoding.java:290)
at java.lang.String.getBytes(String.java:954)
at
com.sun.org.apache.xml.internal.serializer.EncodingInfo.inEncoding(EncodingInfo.java:413)
at
com.sun.org.apache.xml.internal.serializer.EncodingInfo.access$100(EncodingInfo.java:62)
at
com.sun.org.apache.xml.internal.serializer.EncodingInfo$EncodingImpl.isInEncoding(EncodingInfo.java:205)
at
com.sun.org.apache.xml.internal.serializer.EncodingInfo$EncodingImpl.isInEncoding(EncodingInfo.java:181)
at
com.sun.org.apache.xml.internal.serializer.EncodingInfo$EncodingImpl.isInEncoding(EncodingInfo.java:181)
[...SNIP...]
at
com.sun.org.apache.xml.internal.serializer.EncodingInfo$EncodingImpl.isInEncoding(EncodingInfo.java:181)

Has anyone ever seen this before?

Thanks,
Omair

Re: What's up with jdk/test/TestEnv.java?

2009-02-02T18:10:10Z

On 02/02/09 01:37 AM, Alan Bateman wrote:

> Martin Buchholz wrote:
>> I've been trying to get the jtreg tests
>> currently failing with UnknownHostException
>> to start passing instead.
>>
>> I looked at jdk/test/TestEnv.java
>> which looks perfectly designed for our needs.
>>
>> But it's not in openjdk7...!?!?
>> (normally I would expect to see things the other way around)
>> In openjdk7 I just see the hostnames hardcoded into
>> jdk/test/java/nio/channels/TestUtil.java
>>
>> Do y'all plan to forward-port jdk/test/TestEnv.java to openjdk7?
>>
>> Thanks,
>>
>> Martin
>>
> TestEnv was my attempt to centralize the host dependencies so that
> they could be specified to jtreg as a properties file (and make it a
> bit easier to run them outside of Sun's network). Joe was anxious we
> do something about this in 6open so that is why it was pushed there
> first (back in 6open-b12). It's on my list to push this to the jdk7/tl
> repository.

For the curious, this is bug 6720349 "(ch) Channels tests depending on
hosts inside Sun."

-Joe

Re: What's up with jdk/test/TestEnv.java?

2009-02-02T01:37:00Z

Martin Buchholz wrote:

> I've been trying to get the jtreg tests
> currently failing with UnknownHostException
> to start passing instead.
>
> I looked at jdk/test/TestEnv.java
> which looks perfectly designed for our needs.
>
> But it's not in openjdk7...!?!?
> (normally I would expect to see things the other way around)
> In openjdk7 I just see the hostnames hardcoded into
> jdk/test/java/nio/channels/TestUtil.java
>
> Do y'all plan to forward-port jdk/test/TestEnv.java to openjdk7?
>
> Thanks,
>
> Martin
>

TestEnv was my attempt to centralize the host dependencies so that they
could be specified to jtreg as a properties file (and make it a bit
easier to run them outside of Sun's network). Joe was anxious we do
something about this in 6open so that is why it was pushed there first
(back in 6open-b12). It's on my list to push this to the jdk7/tl
repository.

-Alan.

What's up with jdk/test/TestEnv.java?

2009-02-01T13:36:58Z

I've been trying to get the jtreg tests
currently failing with UnknownHostException
to start passing instead.

I looked at jdk/test/TestEnv.java
which looks perfectly designed for our needs.

But it's not in openjdk7...!?!?
(normally I would expect to see things the other way around)
In openjdk7 I just see the hostnames hardcoded into
jdk/test/java/nio/channels/TestUtil.java

Do y'all plan to forward-port jdk/test/TestEnv.java to openjdk7?

Thanks,

Martin

Re: Allowing partial jtreg samejvm safe runs

2009-01-14T13:38:12Z

Mark,

Thanks for the idea behind this patch, and for the patch too.

To you and other jtreg users,

I'd be interested in comments regarding the relative merits of
white-listing vs black-listing.

I'd also be interested in comments regarding the feature of
only using the whitelist when running the whole test suite.
An alternative proposal would be to only use the whitelist
in a default mode, where neither -othervm or -samevm is
defined. Specifying either switch would disable the whitelist
(or blacklist?) lookup.

-- Jon

Mark Wielaard wrote:

Re: RFE: compile/fail tag to recognize compiler failures

2008-12-15T17:22:39Z

Mahmood,

This is not possible because of a combination of reasons, but there are
workarounds.

The problem is that jtreg can potentially invoke the compiler in another
JVM, and so it
would require cooperation from the compiler to get the information you
require.
But jtreg is also spec-ed to work with older versions of the compiler,
where we cannot
retrofit any such behavior.

There are two workarounds.

One is to write the test using "@run main" instead of "@compile/fail"
and to use
JSR 199 to drive the compiler. You can get significantly better
control of the compiler
using JSR 199. You can have the code to invoke the compiler in a
library class,
and have the library class put on the the classpath with the @library tag.
You can run this sort of test with something like this
@library ../lib
@build MyCompilerWrapper
@run main MyCompilerWrapper args

The second workaround is to use golden file testing. To avoid dependence on
the localization of error messages, you can use a hidden compiler option
-XDrawDiagnostics, which prints the error message in a "raw"
non-localized form.
You can run this sort of test with
@compile/fail/ref=MyGolden.out -XDrawDiagnostics MyTest.java

You should be able to find examples of both types of test in the langtools
repository. For use of @library, see, for example, javadoc tests and
JavadocTester (sp?)
For use of -XDrawDiagnostics, grep the javac tests.

-- Jon

Mahmood Ali wrote:

> Greetings,
>
> I would like to request changing compile/fail tag semantics a bit.
>
> Currently if the compile fails to compile the provided class, the test
> passes regardless of the reason of the failure. jtreg does not
> distinguish between failures due to compiler reported errors or
> unexpected exceptions in the compiler.
>
> I request that jtreg to recognize these unexpected errors and treat
> them differently from expected errors and not ignore them silently.
>
> Regards,
> Mahmood
>

RFE: compile/fail tag to recognize compiler failures

2008-12-15T17:00:27Z

Greetings,

I would like to request changing compile/fail tag semantics a bit.

Currently if the compile fails to compile the provided class, the test
passes regardless of the reason of the failure. jtreg does not
distinguish between failures due to compiler reported errors or
unexpected exceptions in the compiler.

I request that jtreg to recognize these unexpected errors and treat
them differently from expected errors and not ignore them silently.

Regards,
Mahmood

Re: Allowing partial jtreg samejvm safe runs

2008-11-23T09:57:13Z

Mark,

I like this approach in that we don't mess up the tests with /othervm
keywords
unnecessarily. I think it is worth distinguishing tests which are
inherently
"othervm" from those which are "currently othervm but potentially
samevm with
a bit of effort".

-- Jon

On Nov 23, 2008, at 9:17 AM, Mark Wielaard wrote:

> Hi,
>
> Jonathan Gibbons has been extending jtreg so that tests can be run in
> the same vm. He also made it so that most langtools tests are run that
> way. That brings an enormous speed up. Since running all jdk tests
> takes
> ages it would be great if we could use this same speedup there.
>
> Problem is that -samevm only works if all tests have been adapted to
> behave well (or marked as needing /othervm). So my idea was to add a
> "whitelist mechanism" so you can indicate which (sub)dirs are ready to
> be run with samevm.
>
> Luckily jtreg has mostly been setup for this already. All Actions ask
> their Script whether or not they should be running in an othervm. So
> we
> hook into that to check the scripts location against a whitelist.
>
> I choose the TEST.ROOT file as the location of the whitelist. Since
> that
> is also the root of the test suite. I made the code so that the
> whitelist is only used when running the full test suite from the root.
> That way you can still experiment with -samevm or -othervm when
> running
> subtrees of tests.
>
> Also CCed jdk-regtest@... since that address is mentioned in the
> TEST.ROOT file with an request to contact them if you change
> anything in
> that file.
>
> Below is the implementation as I checked into icedtea6 (patch also
> attached).
>
> 2008-11-23 Mark Wielaard <mark@...>
>
> * test/jtreg/com/sun/javatest/regtest/Main.java
> (createParameters): Set same jvm safe dirs when non-null.
> (getSameJVMSafeDirs): New method that reads samejvmsafe property
> from TEST.ROOT.
> (sameJVMSafeDirs): New private field.
> * test/jtreg/com/sun/javatest/regtest/RegressionParameters.java
> (SAME_JVM_SAFE_DIRS): New static final field.
> (load): Read same jvm safe dirs.
> (save0): Write same jvm safe dirs.
> (getSameJVMSafeDirs): New method.
> (setSameJVMSafeDirs): New method.
> * test/jtreg/com/sun/javatest/regtest/RegressionScript.java
> (run): Set testDirPath.
> (isOtherJVM): Take same jvm safe into account.
> (isSameJVMSafe): New method.
> (testDirPath): New private field.
>
> For icedtea6 I did a quick scan of the openjdk/jdk/test subdirs and
> listed an initial set that work with -samejvm and added a patch to
> augment the TEST.ROOT file (attached).
>
> 2008-11-23 Mark Wielaard <mark@...>
>
> * patches/icedtea-samejvm-safe.patch: New patch.
> * Makefile.am (ICEDTEA_PATCHES): Add new patch.
> (check-jdk): Run jtreg with -samevm.
> * HACKING: Document new patch.
>
> Comments on the idea and the implementation very appreciated.
>
> With this 'make check-jdk' takes a little bit less than 3 hours on my
> machine now. It was almost 3.5 hours, so it already saves a lot of
> time.
> But we need to make much more tests work with samevm to bring the
> total
> test running time down to something an ordinary developer would be
> comfortable with running every day.
>
> We have to see whether or not that is really possible. It seems jdi,
> rmi
> and awt take up a lot of the time needed to run all tests, and I am
> not
> sure those tests can be speed up much.
>
> Cheers,
>
> Mark
> diff -r 9aae858397f9 test/jtreg/com/sun/javatest/regtest/Main.java
> --- a/test/jtreg/com/sun/javatest/regtest/Main.java Fri Nov 21
> 18:35:27 2008 -0500
> +++ b/test/jtreg/com/sun/javatest/regtest/Main.java Sun Nov 23
> 16:56:34 2008 +0100
> @@ -29,6 +29,7 @@
> import java.io.BufferedReader;
> import java.io.BufferedWriter;
> import java.io.File;
> +import java.io.FileInputStream;
> import java.io.FileNotFoundException;
> import java.io.FileReader;
> import java.io.FileWriter;
> @@ -52,6 +53,7 @@
> import java.util.Iterator;
> import java.util.List;
> import java.util.Map;
> +import java.util.Properties;
> import java.util.TreeMap;
>
> import com.sun.javatest.CompositeFilter;
> @@ -1354,6 +1356,10 @@
> if (ignoreKind != null)
> rp.setIgnoreKind(ignoreKind);
>
> + sameJVMSafeDirs = getSameJVMSafeDirs(ts);
> + if (sameJVMSafeDirs != null)
> + rp.setSameJVMSafeDirs(sameJVMSafeDirs);
> +
> return rp;
> } catch (TestSuite.Fault f) {
> f.printStackTrace();
> @@ -1372,6 +1378,35 @@
> } catch (IOException e) {
> return file.getAbsoluteFile();
> }
> + }
> +
> + // Returns directory (prefix) for tests that are same jvm safe
> + // read from amejvmsafe property in TEST.ROOT file. Returning
> null
> + // means all tests are considered same jvm safe. null is returned
> + // when there is no samejvmsafe property, or there was a problem
> + // reading it, and when anything else than the test root was
> given
> + // as test file argument. Meaning that this only returns
> something
> + // non-null if anything was actually specified as same jvm safe
> and
> + // the whole test suite is being tested.
> + private List<String> getSameJVMSafeDirs(File testRoot) {
> + // Only use the same jvm safe dirs when running from the root.
> + if (testFileArgs.size() != 1
> + || !
> canon(testFileArgs.iterator().next()).equals(canon(testRoot)))
> + return null;
> +
> + try {
> + File file = new File(testRoot, "TEST.ROOT");
> + if (file.exists()) {
> + Properties testRootProps = new Properties();
> + testRootProps.load(new FileInputStream(file));
> + String safedirs = testRootProps.getProperty("samejvmsafe");
> + if ((safedirs != null) && (safedirs.trim().length() > 0))
> + return Arrays.asList(StringArray.splitWS(safedirs));
> + }
> + } catch (IOException ioe) {
> + // Bah, then just assume everything is safe.
> + }
> + return null;
> }
>
> private String getRelativePath(File base, File f) {
> @@ -1762,6 +1797,7 @@
> // these args are jtreg extras
> private File baseDirArg;
> private boolean sameJVMFlag;
> + private List<String> sameJVMSafeDirs;
> private JDK jdk;
> private boolean guiFlag;
> private boolean reportOnlyFlag;
> diff -r 9aae858397f9 test/jtreg/com/sun/javatest/regtest/
> RegressionParameters.java
> --- a/test/jtreg/com/sun/javatest/regtest/RegressionParameters.java
> Fri Nov 21 18:35:27 2008 -0500
> +++ b/test/jtreg/com/sun/javatest/regtest/RegressionParameters.java
> Sun Nov 23 16:56:34 2008 +0100
> @@ -139,6 +139,7 @@
> private static final String TEST_JAVA_OPTIONS = ".testJavaOpts";
> private static final String IGNORE = ".ignore";
> private static final String RETAIN_ARGS = ".retain";
> + private static final String SAME_JVM_SAFE_DIRS =
> ".samejvmsafedirs";
>
> @Override
> public void load(Map data, boolean checkChecksum) throws
> Interview.Fault {
> @@ -182,6 +183,10 @@
> v = (String) data.get(prefix + RETAIN_ARGS);
> if (v != null)
>
> setRetainArgs(Arrays.asList(StringArray.splitSeparator("\n", v)));
> +
> + v = (String) data.get(prefix + SAME_JVM_SAFE_DIRS);
> + if (v != null)
> +
> setSameJVMSafeDirs(Arrays.asList(StringArray.splitSeparator("\n",
> v)));
> }
>
> @SuppressWarnings("unchecked")
> @@ -203,6 +208,9 @@
>
> if (jdk != null)
> data.put(prefix + JDK, jdk.getPath());
> +
> + if (sameJVMSafeDirs != null)
> + data.put(prefix + SAME_JVM_SAFE_DIRS,
> StringUtils.join(sameJVMSafeDirs, "\n"));
>
> if (retainArgs != null)
> data.put(prefix + RETAIN_ARGS,
> StringUtils.join(retainArgs, "\n"));
> @@ -592,6 +600,18 @@
> private Pattern retainFilesPattern;
>
> //---------------------------------------------------------------------
> +
> + List<String> getSameJVMSafeDirs() {
> + return sameJVMSafeDirs;
> + }
> +
> + void setSameJVMSafeDirs(List<String> sameJVMSafeDirs) {
> + this.sameJVMSafeDirs= sameJVMSafeDirs;
> + }
> +
> + private List<String> sameJVMSafeDirs;
> +
> +
> //---------------------------------------------------------------------
>
> private static final String PATHSEP =
> System.getProperty("path.separator");
> private static final String LINESEP =
> System.getProperty("line.separator");
> diff -r 9aae858397f9 test/jtreg/com/sun/javatest/regtest/
> RegressionScript.java
> --- a/test/jtreg/com/sun/javatest/regtest/RegressionScript.java Fri
> Nov 21 18:35:27 2008 -0500
> +++ b/test/jtreg/com/sun/javatest/regtest/RegressionScript.java Sun
> Nov 23 16:56:34 2008 +0100
> @@ -69,6 +69,10 @@
> public Status run(String[] argv, TestDescription td,
> TestEnvironment env) {
> if (!(env instanceof RegressionEnvironment))
> throw new AssertionError();
> +
> + String testFilePath = td.getRootRelativePath();
> + int lastSlash = testFilePath.lastIndexOf('/');
> + testDirPath = testFilePath.substring(0, lastSlash);
>
> regEnv = (RegressionEnvironment) env;
> params = regEnv.params;
> @@ -833,7 +837,26 @@
> }
>
> boolean isOtherJVM() {
> - return params.isOtherJVM();
> + boolean samevm = !params.isOtherJVM();
> + if (samevm)
> + return !isSameJVMSafe();
> + else
> + return true;
> + }
> +
> + // Whether the actions of this script can safely run in the
> same jvm.
> + // No same jvm safe dirs given means they are all assumed safe.
> + // If our actions come from a file in a subdir of a safe dir
> that is ok.
> + boolean isSameJVMSafe() {
> + List<String> dirs = params.getSameJVMSafeDirs();
> + if (dirs == null)
> + return true;
> +
> + for (String dir : dirs)
> + if (testDirPath.startsWith(dir))
> + return true;
> +
> + return false;
> }
>
> String getJavaProg() {
> @@ -921,5 +944,6 @@
>
> private RegressionEnvironment regEnv;
> private RegressionParameters params;
> + private String testDirPath;
> }
>
> <icedtea-samejvm-safe.patch>

Re: Allowing partial jtreg samejvm safe runs

2008-11-23T09:37:19Z

On Sun, 2008-11-23 at 18:17 +0100, Mark Wielaard wrote:
> Also CCed jdk-regtest@... since that address is mentioned in the
> TEST.ROOT file with an request to contact them if you change anything in
> that file.

This address bounces. Should we change it to discuss@...?

Cheers,

Mark

Allowing partial jtreg samejvm safe runs

2008-11-23T09:17:54Z

Hi,

Jonathan Gibbons has been extending jtreg so that tests can be run in
the same vm. He also made it so that most langtools tests are run that
way. That brings an enormous speed up. Since running all jdk tests takes
ages it would be great if we could use this same speedup there.

Problem is that -samevm only works if all tests have been adapted to
behave well (or marked as needing /othervm). So my idea was to add a
"whitelist mechanism" so you can indicate which (sub)dirs are ready to
be run with samevm.

Luckily jtreg has mostly been setup for this already. All Actions ask
their Script whether or not they should be running in an othervm. So we
hook into that to check the scripts location against a whitelist.

I choose the TEST.ROOT file as the location of the whitelist. Since that
is also the root of the test suite. I made the code so that the
whitelist is only used when running the full test suite from the root.
That way you can still experiment with -samevm or -othervm when running
subtrees of tests.

Also CCed jdk-regtest@... since that address is mentioned in the
TEST.ROOT file with an request to contact them if you change anything in
that file.

Below is the implementation as I checked into icedtea6 (patch also
attached).

2008-11-23 Mark Wielaard <mark@...>

* test/jtreg/com/sun/javatest/regtest/Main.java
(createParameters): Set same jvm safe dirs when non-null.
(getSameJVMSafeDirs): New method that reads samejvmsafe property
from TEST.ROOT.
(sameJVMSafeDirs): New private field.
* test/jtreg/com/sun/javatest/regtest/RegressionParameters.java
(SAME_JVM_SAFE_DIRS): New static final field.
(load): Read same jvm safe dirs.
(save0): Write same jvm safe dirs.
(getSameJVMSafeDirs): New method.
(setSameJVMSafeDirs): New method.
* test/jtreg/com/sun/javatest/regtest/RegressionScript.java
(run): Set testDirPath.
(isOtherJVM): Take same jvm safe into account.
(isSameJVMSafe): New method.
(testDirPath): New private field.

For icedtea6 I did a quick scan of the openjdk/jdk/test subdirs and
listed an initial set that work with -samejvm and added a patch to
augment the TEST.ROOT file (attached).

2008-11-23 Mark Wielaard <mark@...>

* patches/icedtea-samejvm-safe.patch: New patch.
* Makefile.am (ICEDTEA_PATCHES): Add new patch.
(check-jdk): Run jtreg with -samevm.
* HACKING: Document new patch.

Comments on the idea and the implementation very appreciated.

With this 'make check-jdk' takes a little bit less than 3 hours on my
machine now. It was almost 3.5 hours, so it already saves a lot of time.
But we need to make much more tests work with samevm to bring the total
test running time down to something an ordinary developer would be
comfortable with running every day.

We have to see whether or not that is really possible. It seems jdi, rmi
and awt take up a lot of the time needed to run all tests, and I am not
sure those tests can be speed up much.

Cheers,

Mark

diff -r 9aae858397f9 test/jtreg/com/sun/javatest/regtest/Main.java
--- a/test/jtreg/com/sun/javatest/regtest/Main.java Fri Nov 21 18:35:27 2008 -0500
+++ b/test/jtreg/com/sun/javatest/regtest/Main.java Sun Nov 23 16:56:34 2008 +0100
@@ -29,6 +29,7 @@
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
+import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.FileWriter;
@@ -52,6 +53,7 @@
import java.util.Iterator;
import java.util.List;
import java.util.Map;
+import java.util.Properties;
import java.util.TreeMap;

import com.sun.javatest.CompositeFilter;
@@ -1354,6 +1356,10 @@
if (ignoreKind != null)
rp.setIgnoreKind(ignoreKind);

+ sameJVMSafeDirs = getSameJVMSafeDirs(ts);
+ if (sameJVMSafeDirs != null)
+ rp.setSameJVMSafeDirs(sameJVMSafeDirs);
+
return rp;
} catch (TestSuite.Fault f) {
f.printStackTrace();
@@ -1372,6 +1378,35 @@
} catch (IOException e) {
return file.getAbsoluteFile();
}
+ }
+
+ // Returns directory (prefix) for tests that are same jvm safe
+ // read from amejvmsafe property in TEST.ROOT file. Returning null
+ // means all tests are considered same jvm safe. null is returned
+ // when there is no samejvmsafe property, or there was a problem
+ // reading it, and when anything else than the test root was given
+ // as test file argument. Meaning that this only returns something
+ // non-null if anything was actually specified as same jvm safe and
+ // the whole test suite is being tested.
+ private List<String> getSameJVMSafeDirs(File testRoot) {
+ // Only use the same jvm safe dirs when running from the root.
+ if (testFileArgs.size() != 1
+ || !canon(testFileArgs.iterator().next()).equals(canon(testRoot)))
+ return null;
+
+ try {
+ File file = new File(testRoot, "TEST.ROOT");
+ if (file.exists()) {
+ Properties testRootProps = new Properties();
+ testRootProps.load(new FileInputStream(file));
+ String safedirs = testRootProps.getProperty("samejvmsafe");
+ if ((safedirs != null) && (safedirs.trim().length() > 0))
+ return Arrays.asList(StringArray.splitWS(safedirs));
+ }
+ } catch (IOException ioe) {
+ // Bah, then just assume everything is safe.
+ }
+ return null;
}

private String getRelativePath(File base, File f) {
@@ -1762,6 +1797,7 @@
// these args are jtreg extras
private File baseDirArg;
private boolean sameJVMFlag;
+ private List<String> sameJVMSafeDirs;
private JDK jdk;
private boolean guiFlag;
private boolean reportOnlyFlag;
diff -r 9aae858397f9 test/jtreg/com/sun/javatest/regtest/RegressionParameters.java
--- a/test/jtreg/com/sun/javatest/regtest/RegressionParameters.java Fri Nov 21 18:35:27 2008 -0500
+++ b/test/jtreg/com/sun/javatest/regtest/RegressionParameters.java Sun Nov 23 16:56:34 2008 +0100
@@ -139,6 +139,7 @@
private static final String TEST_JAVA_OPTIONS = ".testJavaOpts";
private static final String IGNORE = ".ignore";
private static final String RETAIN_ARGS = ".retain";
+ private static final String SAME_JVM_SAFE_DIRS = ".samejvmsafedirs";

@Override
public void load(Map data, boolean checkChecksum) throws Interview.Fault {
@@ -182,6 +183,10 @@
v = (String) data.get(prefix + RETAIN_ARGS);
if (v != null)
setRetainArgs(Arrays.asList(StringArray.splitSeparator("\n", v)));
+
+ v = (String) data.get(prefix + SAME_JVM_SAFE_DIRS);
+ if (v != null)
+ setSameJVMSafeDirs(Arrays.asList(StringArray.splitSeparator("\n", v)));
}

@SuppressWarnings("unchecked")
@@ -203,6 +208,9 @@

if (jdk != null)
data.put(prefix + JDK, jdk.getPath());
+
+ if (sameJVMSafeDirs != null)
+ data.put(prefix + SAME_JVM_SAFE_DIRS, StringUtils.join(sameJVMSafeDirs, "\n"));

if (retainArgs != null)
data.put(prefix + RETAIN_ARGS, StringUtils.join(retainArgs, "\n"));
@@ -592,6 +600,18 @@
private Pattern retainFilesPattern;

//---------------------------------------------------------------------
+
+ List<String> getSameJVMSafeDirs() {
+ return sameJVMSafeDirs;
+ }
+
+ void setSameJVMSafeDirs(List<String> sameJVMSafeDirs) {
+ this.sameJVMSafeDirs= sameJVMSafeDirs;
+ }
+
+ private List<String> sameJVMSafeDirs;
+
+ //---------------------------------------------------------------------

private static final String PATHSEP = System.getProperty("path.separator");
private static final String LINESEP = System.getProperty("line.separator");
diff -r 9aae858397f9 test/jtreg/com/sun/javatest/regtest/RegressionScript.java
--- a/test/jtreg/com/sun/javatest/regtest/RegressionScript.java Fri Nov 21 18:35:27 2008 -0500
+++ b/test/jtreg/com/sun/javatest/regtest/RegressionScript.java Sun Nov 23 16:56:34 2008 +0100
@@ -69,6 +69,10 @@
public Status run(String[] argv, TestDescription td, TestEnvironment env) {
if (!(env instanceof RegressionEnvironment))
throw new AssertionError();
+
+ String testFilePath = td.getRootRelativePath();
+ int lastSlash = testFilePath.lastIndexOf('/');
+ testDirPath = testFilePath.substring(0, lastSlash);

regEnv = (RegressionEnvironment) env;
params = regEnv.params;
@@ -833,7 +837,26 @@
}

boolean isOtherJVM() {
- return params.isOtherJVM();
+ boolean samevm = !params.isOtherJVM();
+ if (samevm)
+ return !isSameJVMSafe();
+ else
+ return true;
+ }
+
+ // Whether the actions of this script can safely run in the same jvm.
+ // No same jvm safe dirs given means they are all assumed safe.
+ // If our actions come from a file in a subdir of a safe dir that is ok.
+ boolean isSameJVMSafe() {
+ List<String> dirs = params.getSameJVMSafeDirs();
+ if (dirs == null)
+ return true;
+
+ for (String dir : dirs)
+ if (testDirPath.startsWith(dir))
+ return true;
+
+ return false;
}

String getJavaProg() {
@@ -921,5 +944,6 @@

private RegressionEnvironment regEnv;
private RegressionParameters params;
+ private String testDirPath;
}

[icedtea-samejvm-safe.patch]

--- openjdk.orig/jdk/test/TEST.ROOT 2008-11-06 09:44:52.000000000 +0100 +++ openjdk/jdk/test/TEST.ROOT 2008-11-23 16:15:53.000000000 +0100 @@ -4,3 +4,9 @@ # The list of keywords supported in the entire test suite keys=2d dnd i18n + +# List if directory (prefixes) that contain test that are safe to run in +# -samejvm mode. Only used if the test root (dir containing this file) is +# given to jtreg and the -samejvm argument is used, ignored otherwise +# (meaning all tests are assumed to be same jvm safe). +samejvmsafe=com/sun/crypto com/sun/management com/sun/media com/sun/security java/beans/beancontext java/beans/PropertyChangeSupport java/beans/Statement java/beans/VetoableChangeSupport java/lang/Boolean java/lang/Byte java/lang/Double java/lang/Float java/lang/Integer java/lang/Long java/lang/Short java/lang/Math java/lang/StrictMath java/lang/String java/lang/Throwable java/lang/instrument java/math java/net/URI java/net/URLDecoder java/net/URLEncoder java/nio/Buffer java/nio/ByteOrder java/nio/MappedByteBuffer java/nio/channels/Channels java/nio/channels/DatagramChannel java/nio/channels/spi java/security/cert java/security/Provider java/text java/util/jar java/util/zip java/util/Array java/util/BitSet java/util/Collection java/util/List javax/crypto/Cipher javax/imageio javax/management/openmbean javax/naming javax/rmi javax/sound sun/net/www/protocol sun/misc sun/nio sun/rmi sun/security/pkcs11 sun/security/rsa sun/security/util sun/security/x509 sun/util

New jtreg version, 4.0 b02

2008-10-21T14:45:20Z

We've released a new build of jtreg, which fixes a number of minor issues.
See my blog entry at http://blogs.sun.com/jjg/entry/jtreg_update for
details.

Thanks to those folk that reported issues and made suggestions, and to
Xiomara for helping finally get the bits published.

-- Jon

Re: Setting PATH for jtreg

2008-10-14T06:50:36Z

Volker,

Thank you for the update and suggestions.

Your comment about it being more of a general option
and not a JDK option and your suggestion about
the FAQ are noted.

-- Jon

On Oct 14, 2008, at 12:30 AM, Volker Simonis wrote:

> Hi John,
>
> thank you for the fast help. "-e:PATH" indeed solved the problem.
>
> Now that you pointed me to the option I must admit that it is really
> there. Unfortunately, I didn't found it before, although I did "jtreg
> -help" several times. This may be perhaps because it is in the
> "JDK-related Options". For me, this is not strictly a JDK-related
> option and would better fit into the "General Options". But this is of
> course purely a matter of taste:)
>
> However I think a small advice in the FAQ section 4.1 cited below that
> PATH can be overriden with the "-e:PATH" option may be helpfull to
> others as well.
>
> Once again thank you and best regards,
> Volker
>
>
> On 10/13/08, Jonathan Gibbons <Jonathan.Gibbons@...> wrote:
>> Volker,
>>
>> jtreg has a -e option for tunnelling env variables into scripts.
>> Use "jtreg -help -e" for info. Any values specified with -e
>> override any defaults set up for the test. So, it should work
>> to use -e:PATH to override the system default value of PATH
>> with the value from your current environment.
>>
>> -- Jon
>>
>>
>>
>> On Oct 13, 2008, at 8:23 AM, Volker Simonis wrote:
>>
>>
>>> Hi,
>>>
>>> the jtreg FAQ states in point 4.1:
>>>
>>> As per spec, the only shell environment variables that are
>>> propagated
>>> into the test's JVM are:
>>>
>>> * Linux and Solaris:
>>> o PATH is set to /bin:/usr/bin
>>>
>>> However in our environment, the compilers are installed on a global
>>> NFS-share. I have the problem that all the tests which use gcc to
>>> compile native JNI files fail because they get a wrong version of
>>> gcc
>>> from /usr/bin.
>>>
>>> Is there any possibility how I can override the PATH variable for
>>> jtreg?
>>>
>>> Regards,
>>> Volker
>>>
>>
>>

Re: Setting PATH for jtreg

2008-10-14T00:30:15Z

Hi John,

thank you for the fast help. "-e:PATH" indeed solved the problem.

Now that you pointed me to the option I must admit that it is really
there. Unfortunately, I didn't found it before, although I did "jtreg
-help" several times. This may be perhaps because it is in the
"JDK-related Options". For me, this is not strictly a JDK-related
option and would better fit into the "General Options". But this is of
course purely a matter of taste:)

However I think a small advice in the FAQ section 4.1 cited below that
PATH can be overriden with the "-e:PATH" option may be helpfull to
others as well.

Once again thank you and best regards,
Volker

On 10/13/08, Jonathan Gibbons <Jonathan.Gibbons@...> wrote:

> Volker,
>
> jtreg has a -e option for tunnelling env variables into scripts.
> Use "jtreg -help -e" for info. Any values specified with -e
> override any defaults set up for the test. So, it should work
> to use -e:PATH to override the system default value of PATH
> with the value from your current environment.
>
> -- Jon
>
>
>
> On Oct 13, 2008, at 8:23 AM, Volker Simonis wrote:
>
>
> > Hi,
> >
> > the jtreg FAQ states in point 4.1:
> >
> > As per spec, the only shell environment variables that are propagated
> > into the test's JVM are:
> >
> > * Linux and Solaris:
> > o PATH is set to /bin:/usr/bin
> >
> > However in our environment, the compilers are installed on a global
> > NFS-share. I have the problem that all the tests which use gcc to
> > compile native JNI files fail because they get a wrong version of gcc
> > from /usr/bin.
> >
> > Is there any possibility how I can override the PATH variable for jtreg?
> >
> > Regards,
> > Volker
> >
>
>

Re: Setting PATH for jtreg

2008-10-13T09:30:42Z

Volker,

jtreg has a -e option for tunnelling env variables into scripts.
Use "jtreg -help -e" for info. Any values specified with -e
override any defaults set up for the test. So, it should work
to use -e:PATH to override the system default value of PATH
with the value from your current environment.

-- Jon

On Oct 13, 2008, at 8:23 AM, Volker Simonis wrote:

> Hi,
>
> the jtreg FAQ states in point 4.1:
>
> As per spec, the only shell environment variables that are propagated
> into the test's JVM are:
>
> * Linux and Solaris:
> o PATH is set to /bin:/usr/bin
>
> However in our environment, the compilers are installed on a global
> NFS-share. I have the problem that all the tests which use gcc to
> compile native JNI files fail because they get a wrong version of gcc
> from /usr/bin.
>
> Is there any possibility how I can override the PATH variable for
> jtreg?
>
> Regards,
> Volker

Setting PATH for jtreg

2008-10-13T08:23:17Z

Hi,

the jtreg FAQ states in point 4.1:

As per spec, the only shell environment variables that are propagated
into the test's JVM are:

* Linux and Solaris:
o PATH is set to /bin:/usr/bin

However in our environment, the compilers are installed on a global
NFS-share. I have the problem that all the tests which use gcc to
compile native JNI files fail because they get a wrong version of gcc
from /usr/bin.

Is there any possibility how I can override the PATH variable for jtreg?

Regards,
Volker

Re: More careful identification of @ignore

2008-10-02T12:26:41Z

I'm sorry; that is ... disappointing. I'll raise the priority to fix this.

-- Jon

Martin Buchholz wrote:

On Thu, Oct 2, 2008 at 11:47, Jonathan Gibbons Jonathan.Gibbons@... wrote:

Egads, 'twould appear t'be true.

As a workaround, I suggest you use the -ignore switch (e.g. -ignore:quiet)
instead of -k:\!ignore.


I tried -ignore:quiet,
but that appears to have the same bug;
that is, it is not a workaround.

Martin

Re: More careful identification of @ignore

2008-10-02T12:17:53Z

On Thu, Oct 2, 2008 at 11:47, Jonathan Gibbons <Jonathan.Gibbons@...> wrote:
> Egads, 'twould appear t'be true.
>
> As a workaround, I suggest you use the -ignore switch (e.g. -ignore:quiet)
> instead of -k:\!ignore.

I tried -ignore:quiet,
but that appears to have the same bug;
that is, it is not a workaround.

Martin

Re: More careful identification of @ignore

2008-10-02T11:47:37Z

Egads, 'twould appear t'be true.

As a workaround, I suggest you use the -ignore switch (e.g. -ignore:quiet)
instead of -k:\!ignore.

-- Jon

Jonathan Gibbons wrote:

> Eeeek, so noted. I'll investigate.
>
> -- Jon
>
>
> Martin Buchholz wrote:
>> Hi jtreg maintainers,
>>
>> This is a bug report.
>>
>> If I try to test jdk/test/sun/net/idn/PunycodeTest.java,
>> with @ignore'd tests suppressed, I get:
>>
>> jdk/test/sun/net/idn $ ~/jct-tools/3.2.2_03/linux/bin/jtreg
>> -v:nopass,fail -automatic "-k:\!ignore"
>> -testjdk:$FOREST/jdk/../build/linux-i586/j2sdk-image PunycodeTest.java
>> Test results: no tests selected
>>
>> This is due to the occurrence of the string "ignore" in
>>
>> * @compile -XDignore.symbol.file PunycodeTest.java
>>
>> as any mangling of that string in the sources and retesting
>> demonstrates.
>>
>> Probably the original coder was lazy and simply searched for the
>> "ignore" substring,
>> whereas a more disciplined search for @ignore or maybe @key WORD ...
>> is necessary.
>>
>> Martin
>>
>
>

Re: More careful identification of @ignore

2008-10-02T11:29:13Z

Eeeek, so noted. I'll investigate.

-- Jon

Martin Buchholz wrote:

> Hi jtreg maintainers,
>
> This is a bug report.
>
> If I try to test jdk/test/sun/net/idn/PunycodeTest.java,
> with @ignore'd tests suppressed, I get:
>
> jdk/test/sun/net/idn $ ~/jct-tools/3.2.2_03/linux/bin/jtreg
> -v:nopass,fail -automatic "-k:\!ignore"
> -testjdk:$FOREST/jdk/../build/linux-i586/j2sdk-image PunycodeTest.java
> Test results: no tests selected
>
> This is due to the occurrence of the string "ignore" in
>
> * @compile -XDignore.symbol.file PunycodeTest.java
>
> as any mangling of that string in the sources and retesting demonstrates.
>
> Probably the original coder was lazy and simply searched for the
> "ignore" substring,
> whereas a more disciplined search for @ignore or maybe @key WORD ...
> is necessary.
>
> Martin
>

More careful identification of @ignore

2008-10-02T11:14:17Z

Hi jtreg maintainers,

This is a bug report.

If I try to test jdk/test/sun/net/idn/PunycodeTest.java,
with @ignore'd tests suppressed, I get:

jdk/test/sun/net/idn $ ~/jct-tools/3.2.2_03/linux/bin/jtreg
-v:nopass,fail -automatic "-k:\!ignore"
-testjdk:$FOREST/jdk/../build/linux-i586/j2sdk-image PunycodeTest.java
Test results: no tests selected

This is due to the occurrence of the string "ignore" in

* @compile -XDignore.symbol.file PunycodeTest.java

as any mangling of that string in the sources and retesting demonstrates.

Probably the original coder was lazy and simply searched for the
"ignore" substring,
whereas a more disciplined search for @ignore or maybe @key WORD ...
is necessary.

Martin

Re: Reporting into hudson

2008-09-25T16:52:22Z

Well, the initial experiment to enable XML reports is clearly easy.

-- Jon

On Sep 25, 2008, at 4:07 PM, David Herron wrote:

> sheesh, I clicked SEND a moment too soon..
>
> It occurred to me the real request may not be javatest-compatible
> XML reports as that might not work out right, and it might be
> necessary to instead teach hudson how to understand jtreg results.
>
>
> - David
>
>
> David Herron wrote:
>> Oh, I see.
>> So, pretty please, with sugar on top, I'd love it if jtreg were to
>> produce javatest-compatible XML report files we can display in
>> hudson? ;-)
>>
>> - David
>>
>>
>> Jonathan Gibbons wrote:
>>> Not until someone asks.
>>>
>>> -- Jon
>>>
>>> David Herron wrote:
>>>> Hi, I'm fiddling with running openjdk builds & jtreg under
>>>> hudson ... (see http://home.7gen.com:8080 ..)
>>>>
>>>> What I'm looking at right now is how to get jtreg test results to
>>>> display in hudson.
>>>>
>>>> My question is.. since jtreg is a wrapper around javatest,
>>>> shouldn't there be a way to use the hudson plugin for javatest
>>>> results to make a pretty graph on the hudson dashboard?
>>>>
>>>> I have the javatest plugin installed & enabled, and in the
>>>> configuration it wants the work directory (easy) plus the fileset
>>>> for the XML files. Uh... There isn't a jtreg option to produce a
>>>> report in XML files. Is there?
>>>> - David Herron
>>>>
>>>
>>
>

Re: Reporting into hudson

2008-09-25T16:07:55Z

sheesh, I clicked SEND a moment too soon..

It occurred to me the real request may not be javatest-compatible XML
reports as that might not work out right, and it might be necessary to
instead teach hudson how to understand jtreg results.

- David

David Herron wrote:

> Oh, I see.
> So, pretty please, with sugar on top, I'd love it if jtreg were to
> produce javatest-compatible XML report files we can display in hudson?
> ;-)
>
> - David
>
>
> Jonathan Gibbons wrote:
>> Not until someone asks.
>>
>> -- Jon
>>
>> David Herron wrote:
>>> Hi, I'm fiddling with running openjdk builds & jtreg under hudson
>>> ... (see http://home.7gen.com:8080 ..)
>>>
>>> What I'm looking at right now is how to get jtreg test results to
>>> display in hudson.
>>>
>>> My question is.. since jtreg is a wrapper around javatest, shouldn't
>>> there be a way to use the hudson plugin for javatest results to make
>>> a pretty graph on the hudson dashboard?
>>>
>>> I have the javatest plugin installed & enabled, and in the
>>> configuration it wants the work directory (easy) plus the fileset
>>> for the XML files. Uh... There isn't a jtreg option to produce a
>>> report in XML files. Is there?
>>> - David Herron
>>>
>>
>

Re: Reporting into hudson

2008-09-25T16:03:10Z

Oh, I see.

So, pretty please, with sugar on top, I'd love it if jtreg were to
produce javatest-compatible XML report files we can display in hudson? ;-)

- David

Jonathan Gibbons wrote:

> Not until someone asks.
>
> -- Jon
>
> David Herron wrote:
>> Hi, I'm fiddling with running openjdk builds & jtreg under hudson ...
>> (see http://home.7gen.com:8080 ..)
>>
>> What I'm looking at right now is how to get jtreg test results to
>> display in hudson.
>>
>> My question is.. since jtreg is a wrapper around javatest, shouldn't
>> there be a way to use the hudson plugin for javatest results to make
>> a pretty graph on the hudson dashboard?
>>
>> I have the javatest plugin installed & enabled, and in the
>> configuration it wants the work directory (easy) plus the fileset for
>> the XML files. Uh... There isn't a jtreg option to produce a report
>> in XML files. Is there?
>> - David Herron
>>
>

Re: Reporting into hudson

2008-09-25T15:53:18Z

Not until someone asks.

-- Jon

David Herron wrote:

> Hi, I'm fiddling with running openjdk builds & jtreg under hudson ...
> (see http://home.7gen.com:8080 ..)
>
> What I'm looking at right now is how to get jtreg test results to
> display in hudson.
>
> My question is.. since jtreg is a wrapper around javatest, shouldn't
> there be a way to use the hudson plugin for javatest results to make a
> pretty graph on the hudson dashboard?
>
> I have the javatest plugin installed & enabled, and in the
> configuration it wants the work directory (easy) plus the fileset for
> the XML files. Uh... There isn't a jtreg option to produce a report
> in XML files. Is there?
> - David Herron
>

Reporting into hudson

2008-09-25T15:43:40Z

Hi, I'm fiddling with running openjdk builds & jtreg under hudson ...
(see http://home.7gen.com:8080 ..)

What I'm looking at right now is how to get jtreg test results to
display in hudson.

My question is.. since jtreg is a wrapper around javatest, shouldn't
there be a way to use the hudson plugin for javatest results to make a
pretty graph on the hudson dashboard?

I have the javatest plugin installed & enabled, and in the configuration
it wants the work directory (easy) plus the fileset for the XML files.
Uh... There isn't a jtreg option to produce a report in XML files. Is
there?

- David Herron

Re: jtreg testing integrated

2008-05-22T15:04:47Z

Hi Kelly,

On Thu, 2008-05-22 at 11:59 -0700, Kelly O'Hair wrote:

> Mark Wielaard wrote:
> > I am not sure. It does take about 3 hours to run all the included tests
> > (and I assume that when we add more tests or integrate things like mauve
> > it will rise). But I do hope people, not just integrators, will run them
> > regularly. Especially when they are working on/integrating larger
> > patches. And we can always fall back on autobuilders so we have a full
> > report at least soon after something bad happens so there is some chance
> > to revert a change relatively quickly.
>
> 3 hours for runs with:
> -client and -server?
> one OS?
> 32bit and 64bit?
>
> And you are only talking about the tests in the jdk/test area I assume.

No all the jtreg based test (-a -ignore:quiet) currently included in
langtools, jdk and hotspot. On a x86_64, dual core, 3.2Ghz, Fedora 8.

> The issue I have seen with the testing is that if it isn't done on a good
> spread of options and platforms and variations, something gets missed,
> there is little consistency between developers as to what the official
> requires test matrix is. Once we have a 100% pass list, and a required
> matrix, I have a system to help enforce these tests being run, but so
> far, nobody has given me a way to run just that list.
> Once I have it, I think we can keep all or most repositories golden.

Yes, that would be ideal.

> The Hotspot team runs many many variations, using options like -Xcomp
> and -Xbatch and loads more. But they are trying to stress the VM.

I assume there are many more hotspot tests than the 4 currently
included. Hopefully they can be liberated so more people can run them.

> I'm more of a 'test before integrate' person, with streamlining and
> automating the testing process, making it part of the developer push process,
> adapting the tests as major regressions sneak by (you can never catch all
> regressions, no matter what you do). Blocking pushes on any failure.
> So I'm trying to throw hardware at the problem until we can possible do
> the "exhaustive testing" that Martin mentions, as part of a developer
> pushing a change in, before anyone else sees it, all automated.

With a more distributed version control system a lot more can be
separated I guess. Your idea of a core test of tests that should always
pass 100% is good. Then autobuilders could take over. And everybody that
cares about a particular architecture/setup/configuration could add
their own autobuilder to the mix and make sure the full blown testsuite
keeps passing completely.

For GCC there is a nice system where when someone commits something on a
platform that they don't have access to a build machine runs and send
email to that person "After your latest commit the gnats Ada compiler
cross compiled from mips64 to ppc AIX failed the following tests. GO FIX
IT!".

> But for automation, I want a test guarantee:
> "these tests should pass 100% of the time on all platforms in all situations"
> and then we can think about enforcing it.
> None of this wishy washy "this test fails sometimes due to the phase of
> the moon" crap. ... oops, can I say crap in a public email??... ;^}

Yes, that is the biggest danger "flaky tests". With Mauve we actually
have that problem. And we are constantly fighting it. It is a huge cost
to all involved :{

Cheers,

Mark