Nabble - OpenJDK Serviceability Development

Re: Need reviewers - 3 bugs

2009-12-10T16:30:24Z

Brad had to physically demonstrate this to me, I thought maybe I
had some kind of brain injury or something for not understanding it. :^(

Anyway, go to the "New" bug and you can see the components, click
on a component and it will show you who the component is assigned to,
e.g. watch-serviceability-other "at" bugs.openjdk.java.net

Go to your "Preferences->Email Preferences" and copy that email address
in as a "user" into the:
"Add users to my watch list (comma separated list):"
field, then Submit.

So completely obvious <--- sarcasm
don't know what was wrong with me. ;^)

-kto

David Holmes - Sun Microsystems wrote:

> I can't get it to work. Where are the "components" listed? Even
> watch-serviceability-other reported "did not match anything" :(
>
> Otherwise great info, but I sure wish it was a lot easier to manage.
>
> Thanks,
> David
>
> Kelly O'Hair said the following on 12/11/09 09:28:
>> Thanks Tim. Brad just explained this to me.
>>
>> I'm setup to watch serviceability and build bugs now.
>>
>> -kto
>>
>> Tim Bell wrote:
>>> Kelly O'Hair wrote:
>>>
>>>> How could I at least see some email on these bugs?
>>>> Aren't they connected to the various openjdk email aliases?
>>>>
>>>> Sorry I'm not bugzilla-savy. :^(
>>>
>>> Visit http://bugs.openjdk.java.net/ and log into your bugzilla account
>>> (Sun employee logins are probably firstname.lastname@..., unless
>>> you set yours up differently)
>>>
>>> Visit your Preferences page and click the [Email Preferences] tab:
>>>
>>> https://bugs.openjdk.java.net/userprefs.cgi?tab=email
>>>
>>> Scroll down to "User Watching"
>>>
>>> Quoting from the OpenJDK Bugzilla announcement page:
>>>> There are Bugzilla "watcher" email accounts set up for each
>>>> component. (e.g. watch-product-component at bugs dot openjdk dot
>>>> java dot net) If you would like to watch for updates to specific
>>>> components, set your email watch preferences accordingly.
>>>
>>> Here are two that I watch (obfuscated by changing '@' to ' at '):
>>>
>>> watch-build-other at bugs.openjdk.java.net
>>> watch-serviceability-other at bugs.openjdk.net
>>>
>>>
>>> Open a second browser window, go to the basic search page:
>>> https://bugs.openjdk.java.net/query.cgi
>>>
>>> Look at the "Product"s listed.
>>>
>>> For example, if you wanted to watch the first product shown (2d)
>>> switch back to your Email Preferences and put in:
>>>
>>> watch-2d-other at bugs.openjdk.java.net
>>>
>>> (Remember to undo the @ obfuscation)
>>>
>>> Hope this helps-
>>> Tim

Re: Need reviewers - 3 bugs

2009-12-10T15:54:47Z

I can't get it to work. Where are the "components" listed? Even
watch-serviceability-other reported "did not match anything" :(

Otherwise great info, but I sure wish it was a lot easier to manage.

Thanks,
David

Kelly O'Hair said the following on 12/11/09 09:28:

> Thanks Tim. Brad just explained this to me.
>
> I'm setup to watch serviceability and build bugs now.
>
> -kto
>
> Tim Bell wrote:
>> Kelly O'Hair wrote:
>>
>>> How could I at least see some email on these bugs?
>>> Aren't they connected to the various openjdk email aliases?
>>>
>>> Sorry I'm not bugzilla-savy. :^(
>>
>> Visit http://bugs.openjdk.java.net/ and log into your bugzilla account
>> (Sun employee logins are probably firstname.lastname@..., unless
>> you set yours up differently)
>>
>> Visit your Preferences page and click the [Email Preferences] tab:
>>
>> https://bugs.openjdk.java.net/userprefs.cgi?tab=email
>>
>> Scroll down to "User Watching"
>>
>> Quoting from the OpenJDK Bugzilla announcement page:
>>> There are Bugzilla "watcher" email accounts set up for each
>>> component. (e.g. watch-product-component at bugs dot openjdk dot java
>>> dot net) If you would like to watch for updates to specific
>>> components, set your email watch preferences accordingly.
>>
>> Here are two that I watch (obfuscated by changing '@' to ' at '):
>>
>> watch-build-other at bugs.openjdk.java.net
>> watch-serviceability-other at bugs.openjdk.net
>>
>>
>> Open a second browser window, go to the basic search page:
>> https://bugs.openjdk.java.net/query.cgi
>>
>> Look at the "Product"s listed.
>>
>> For example, if you wanted to watch the first product shown (2d)
>> switch back to your Email Preferences and put in:
>>
>> watch-2d-other at bugs.openjdk.java.net
>>
>> (Remember to undo the @ obfuscation)
>>
>> Hope this helps-
>> Tim

Re: Need reviewers - 3 bugs

2009-12-10T15:28:01Z

Thanks Tim. Brad just explained this to me.

I'm setup to watch serviceability and build bugs now.

-kto

Tim Bell wrote:

> Kelly O'Hair wrote:
>
>> How could I at least see some email on these bugs?
>> Aren't they connected to the various openjdk email aliases?
>>
>> Sorry I'm not bugzilla-savy. :^(
>
> Visit http://bugs.openjdk.java.net/ and log into your bugzilla account
> (Sun employee logins are probably firstname.lastname@..., unless
> you set yours up differently)
>
> Visit your Preferences page and click the [Email Preferences] tab:
>
> https://bugs.openjdk.java.net/userprefs.cgi?tab=email
>
> Scroll down to "User Watching"
>
> Quoting from the OpenJDK Bugzilla announcement page:
>> There are Bugzilla "watcher" email accounts set up for each component. (e.g. watch-product-component at bugs dot openjdk dot java dot net) If you would like to watch for updates to specific components, set your email watch preferences accordingly.
>
> Here are two that I watch (obfuscated by changing '@' to ' at '):
>
> watch-build-other at bugs.openjdk.java.net
> watch-serviceability-other at bugs.openjdk.net
>
>
> Open a second browser window, go to the basic search page:
> https://bugs.openjdk.java.net/query.cgi
>
> Look at the "Product"s listed.
>
> For example, if you wanted to watch the first product shown (2d)
> switch back to your Email Preferences and put in:
>
> watch-2d-other at bugs.openjdk.java.net
>
> (Remember to undo the @ obfuscation)
>
> Hope this helps-
> Tim

Re: Need reviewers - 3 bugs

2009-12-10T15:15:41Z

Kelly O'Hair wrote:

> How could I at least see some email on these bugs?
> Aren't they connected to the various openjdk email aliases?
>
> Sorry I'm not bugzilla-savy. :^(

Visit http://bugs.openjdk.java.net/ and log into your bugzilla account
(Sun employee logins are probably firstname.lastname@..., unless
you set yours up differently)

Visit your Preferences page and click the [Email Preferences] tab:

https://bugs.openjdk.java.net/userprefs.cgi?tab=email

Scroll down to "User Watching"

Quoting from the OpenJDK Bugzilla announcement page:
> There are Bugzilla "watcher" email accounts set up for each component. (e.g. watch-product-component at bugs dot openjdk dot java dot net) If you would like to watch for updates to specific components, set your email watch preferences accordingly.

Here are two that I watch (obfuscated by changing '@' to ' at '):

watch-build-other at bugs.openjdk.java.net
watch-serviceability-other at bugs.openjdk.net

Open a second browser window, go to the basic search page:
https://bugs.openjdk.java.net/query.cgi

Look at the "Product"s listed.

For example, if you wanted to watch the first product shown (2d)
switch back to your Email Preferences and put in:

watch-2d-other at bugs.openjdk.java.net

(Remember to undo the @ obfuscation)

Hope this helps-
Tim

Re: Need reviewers - 3 bugs

2009-12-10T14:57:23Z

Andrew John Hughes wrote:

> 2009/11/18 Kelly O'Hair <Kelly.Ohair@...>:
>> Need a reviewer.
>>
>> The jhat change is from keith.randall@.... I wasn't able to
>> come up with a testcase for that change.
>>
>> The other two just fix a couple of broken tests and remove them from
>> the ProblemList.
>>
>> 6892742: Improve root set used by jhat
>> 6902323: Fix testcase sun/tools/native2ascii/NativeErrors.java
>> 6902325: Fix testcase sun/tools/jhat/HatHeapDump1Test.java
>>
>> http://cr.openjdk.java.net/~ohair/openjdk7/jdk7-tl-jhat-6892742/webrev/
>>
>> -kto
>>
>>
>>
>
> Pretty much the same fix for the JHat issue was filed by one of our
> Red Hat interns in March:
>
> http://bugs.openjdk.java.net/show_bug.cgi?id=100026
>
> but sadly completely ignored by anyone at Sun (at least it seems so
> from the bug report). This is one reason why I've tended to not
> bother with the OpenJDK bug system...
>

Sorry. I already have to deal with Sun's internal bug tracking system,
and also JIRA for JavaFX. Learning my way around bugzilla has not been
high my list. I get lost and tired of waiting every time I get into
the bugzilla pages. :^( I suspect most Sun people may be in the same boat.

How could I at least see some email on these bugs?
Aren't they connected to the various openjdk email aliases?

Sorry I'm not bugzilla-savy. :^(

> Anyway, coming to merge that with the version now in OpenJDK6 version
> I found that the upstream doesn't update the comment. So perhaps we
> could push:
>
> http://cr.openjdk.java.net/~andrew/jhat/webrev.01/
>
> given a bug ID for this small change.

6909442: Fix comments in test/sun/tools/jhat/HatRun.java

-kto

Re: Need reviewers - 3 bugs

2009-12-10T13:36:15Z

2009/11/18 Kelly O'Hair <Kelly.Ohair@...>:

>
> Need a reviewer.
>
> The jhat change is from keith.randall@.... I wasn't able to
> come up with a testcase for that change.
>
> The other two just fix a couple of broken tests and remove them from
> the ProblemList.
>
> 6892742: Improve root set used by jhat
> 6902323: Fix testcase sun/tools/native2ascii/NativeErrors.java
> 6902325: Fix testcase sun/tools/jhat/HatHeapDump1Test.java
>
> http://cr.openjdk.java.net/~ohair/openjdk7/jdk7-tl-jhat-6892742/webrev/
>
> -kto
>
>
>

Pretty much the same fix for the JHat issue was filed by one of our
Red Hat interns in March:

http://bugs.openjdk.java.net/show_bug.cgi?id=100026

but sadly completely ignored by anyone at Sun (at least it seems so
from the bug report). This is one reason why I've tended to not
bother with the OpenJDK bug system...

Anyway, coming to merge that with the version now in OpenJDK6 version
I found that the upstream doesn't update the comment. So perhaps we
could push:

http://cr.openjdk.java.net/~andrew/jhat/webrev.01/

given a bug ID for this small change.
--
Andrew :-)

Free Java Software Engineer
Red Hat, Inc. (http://www.redhat.com)

Support Free Java!
Contribute to GNU Classpath and the OpenJDK
http://www.gnu.org/software/classpath
http://openjdk.java.net

PGP Key: 94EFD9D8 (http://subkeys.pgp.net)
Fingerprint: F8EF F1EA 401E 2E60 15FA 7927 142C 2591 94EF D9D8

hg: jdk7/tl/jdk: 4891262: API spec, javax/accessibility: few invalid javadoc tags

2009-12-10T13:28:47Z

Changeset: aa0d374b40a5
Author: darcy
Date: 2009-12-10 13:28 -0800
URL: http://hg.openjdk.java.net/jdk7/tl/jdk/rev/aa0d374b40a5

4891262: API spec, javax/accessibility: few invalid javadoc tags
Reviewed-by: jjg

! src/share/classes/javax/accessibility/AccessibleContext.java
! src/share/classes/javax/accessibility/AccessibleExtendedText.java
! src/share/classes/javax/accessibility/AccessibleKeyBinding.java

hg: jdk7/tl/jdk: 6909070: Missing package statements in java.text.Bidi @see links

2009-12-10T13:05:37Z

Changeset: f9c9c2b726a0
Author: darcy
Date: 2009-12-10 13:04 -0800
URL: http://hg.openjdk.java.net/jdk7/tl/jdk/rev/f9c9c2b726a0

6909070: Missing package statements in java.text.Bidi @see links
Reviewed-by: anthony

! src/share/classes/java/text/Bidi.java

hg: jdk7/tl/jdk: 2 new changesets

2009-12-10T11:08:32Z

Changeset: ae944611249f
Author: chegar
Date: 2009-12-10 15:52 +0000
URL: http://hg.openjdk.java.net/jdk7/tl/jdk/rev/ae944611249f

6909089: Memory leak occurs by lack of free for read buffer in SocketInputStream#read()
Reviewed-by: alanb, jccollet

! src/windows/native/java/net/SocketInputStream.c

Changeset: 59221d9e0c1c
Author: chegar
Date: 2009-12-10 09:08 -0800
URL: http://hg.openjdk.java.net/jdk7/tl/jdk/rev/59221d9e0c1c

Merge

hg: jdk7/tl/jdk: 2 new changesets

2009-12-10T08:44:46Z

Changeset: 7c9be6c9385a
Author: mullan
Date: 2009-12-10 11:31 -0500
URL: http://hg.openjdk.java.net/jdk7/tl/jdk/rev/7c9be6c9385a

6867348: Digest Value of References inside Manifest - calculation order problem
Reviewed-by: xuelei

! src/share/classes/org/jcp/xml/dsig/internal/dom/DOMXMLSignature.java
! test/javax/xml/crypto/dsig/GenerationTests.java

Changeset: 421e652f00c7
Author: mullan
Date: 2009-12-10 11:34 -0500
URL: http://hg.openjdk.java.net/jdk7/tl/jdk/rev/421e652f00c7

Merge

Re: Allow Java's ELF symtab reader to use separate debuginfo files

2009-12-10T03:16:30Z

Andrew Haley wrote:
> Kelly O'Hair wrote:
>> Is there some other way than baking in 256 hex constants into our sources?
>
> I'm sure it's perfectly possible to generate the CRC-32 tables at runtime.
> However, the CRC function is defined in terms of this exact source code,

Ah, I was about to add "but it's only 1 kbyte!" when I noticed that
the data are unsigned longs, and therefore maybe 2 kbytes. I presume
that this is because gdb might be used on hosts with 16-bit ints, but
that hardly affects us. I'll fix it before submitting a webrev.

Andrew.

hg: jdk7/tl/jdk: 6909082: Docs warning from java.util.logging.PlatformLoggingMXBean

2009-12-09T23:07:12Z

Changeset: 0f8f624ddcb4
Author: mchung
Date: 2009-12-09 21:09 -0800
URL: http://hg.openjdk.java.net/jdk7/tl/jdk/rev/0f8f624ddcb4

6909082: Docs warning from java.util.logging.PlatformLoggingMXBean
Summary: Fix incorrect tag @See with @see.
Reviewed-by: darcy

! src/share/classes/java/util/logging/PlatformLoggingMXBean.java

hg: jdk7/tl/jdk: 6909057: @see Arrays#hashCode missing particular method specification in j.u.Objects.hash

2009-12-09T17:49:30Z

Changeset: c2f83e13bbe5
Author: darcy
Date: 2009-12-09 17:55 -0800
URL: http://hg.openjdk.java.net/jdk7/tl/jdk/rev/c2f83e13bbe5

6909057: @see Arrays#hashCode missing particular method specification in j.u.Objects.hash
Reviewed-by: ksrini

! src/share/classes/java/util/Objects.java

Re: Need reviewers, jdk7 testing changes

2009-12-09T16:04:40Z

Kelly O'Hair wrote:

>
> Martin Buchholz wrote:
>> On Tue, Dec 8, 2009 at 16:42, Kelly O'Hair <Kelly.Ohair@...> wrote:
>>
>>>> In theory, the process is supposed to prevent
>>>> regression test failures from ever creeping in - that's the whole
>>>> point.
>>>> When they do (inevitably) creep in,
>>>> they are supposed to be aggressively targeted.
>>>> A gatekeeper demonstrates the failure to a developer,
>>>> and the developer is given X time units to fix the test breakage,
>>>> or face reversion of the breakage-inducing change.
>>> But it's not that simple. In some cases the cause of the failure is
>>> a change made to a different repository, by a different team,
>>
>> Changes that get in this way are signs of process failure.
>> E.g. failing tests in java.util caused by hotspot commits
>
> Deja Vu... we have had this discussion before. ;^)
>
> How can any team possibly run ALL the tests?
> We would all grind to a halt.
> And what is this "signs of process failure", are you some kind
> of manager now? ;^) "Hurumph... Every failure is a result of poor
> planning..."
> (just yanking your chain. ;^)
>
> Testing has become a balancing act for everyone, we do what
> we can given the constraints we have. When things fall through
> the cracks, we try and patch the cracks so it doesn't happen
> again. We use what we can to automate, but there are limits.

There is a weeks-long process separating when a change gets pushed into
a JDK 7 integration workspace, like TL, and when the promoted bits are
available for download. The closer a changeset gets to the master, the
most tests should be run against it.

An individual developer should not be expected to run all the tests, but
if at least the regression tests in the code base are not all run on
multiple platform by the time the change hits the master, I'd argue
there is a process problem. Of course, it is also a problem if the
tests are run and the results aren't examined adequately.

-Joe

What to do about jvmti.h

2009-12-09T13:31:23Z

Concerning:

6784084: jvmti.h created during the build differs from file in jdk repo and should not

Several issues here, all need to be solved.

1. The jdk repository manages a copy of the hotspot generated file jvmti.h

2. The jdk Makefile sanity checks are trying to verify that the copy of jvmti.h
matches the one created by the hotspot build. This logic is complicated and
currently broken for jdk7.

3. In general, Hotspot express could be used in any jdk release, and we need to
make sure the right jvmti.h is placed in the jdk install image include/
directory. Even if a hotspot can support multiple jvmti.h versions,
we need to make sure the right release of jvmti.h is delivered, to
preserve the interface versioning.
(Or we need some kind of super jvmti.h??? Exercise left to the reader.)

For jdk7, 1 & 2 are easily solved, but that isn't a complete answer, so
consider this webrev just a start:
http://cr.openjdk.java.net/~ohair/openjdk7/jdk7-tl-jvmti-6784084/webrev/

My opinion is that jvmti.h is a hotspot interface, and belongs to hotspot.
But who should control which version of jvmti.h goes into what jdk release?

My above webrev will have the jdk build use and deliver in include/ the
jvmti.h provided by the hotspot build.
We could have the hotspot build be sensitive to the java version being
built and deliver a single correct jvmti.h file, or we could have hotspot
provide a set of jvmti.h files and the jdk makefiles would pick the one
it wants based on the java version.

Other ideas?

-kto

Re: Need reviewers, jdk7 testing changes

2009-12-09T10:30:48Z

Martin Buchholz wrote:

> On Tue, Dec 8, 2009 at 16:42, Kelly O'Hair <Kelly.Ohair@...> wrote:
>
>>> In theory, the process is supposed to prevent
>>> regression test failures from ever creeping in - that's the whole point.
>>> When they do (inevitably) creep in,
>>> they are supposed to be aggressively targeted.
>>> A gatekeeper demonstrates the failure to a developer,
>>> and the developer is given X time units to fix the test breakage,
>>> or face reversion of the breakage-inducing change.
>> But it's not that simple. In some cases the cause of the failure is
>> a change made to a different repository, by a different team,
>
> Changes that get in this way are signs of process failure.
> E.g. failing tests in java.util caused by hotspot commits

Deja Vu... we have had this discussion before. ;^)

How can any team possibly run ALL the tests?
We would all grind to a halt.
And what is this "signs of process failure", are you some kind
of manager now? ;^) "Hurumph... Every failure is a result of poor planning..."
(just yanking your chain. ;^)

Testing has become a balancing act for everyone, we do what
we can given the constraints we have. When things fall through
the cracks, we try and patch the cracks so it doesn't happen
again. We use what we can to automate, but there are limits.

> are signs the hotspot commit was inadequately tested.

I'll let you arm wrestle the hotspot guys over that comment. ;^)

>
>> or even to the base system software via an update.
>> The ProblemList.txt file was meant to deal with that situation,
>> and also tests that are spuriously failing for unknown reasons.
>
> I agree that flaky and platform-dependent failures are a big problem.
>
>> When the gatekeeper can do what you say, that is great, and ideal.
>> But I just don't see it happening that way in all situations.
>>
>>> I would like to see more effort devoted to fixing the tests
>>> (or the code!) rather than adding infrastructure that might
>>> have the effect of hiding the test failures.
>> Sigh... I'm not trying to hide the failures, and if you haven't
>> noticed, I have fixed quite a few tests myself.
>
> I have noticed, and I do appreciate it.
>
> I do some flaky test fixing myself.

Well to your credit, tests you have worked on in the past are
typically not on my flaky list.
Unfortunately, some of your tests are damn good tests, and cut
a larger swath through the jdk logic than what I suspect your
original test focus was. So your tests may come under more
scrutiny as bugs in various areas get filed, pointing at your
tests and the nasty little test that found their bug. ;^)

>
>> If anything, I'm making the fact that we have test failures more
>> public, and the ultimate goal is to fix the tests.
>> But I needed a baseline, a line in the sand, an expectation
>> of what tests should always pass. And an ability to run all
>> the tests in a timely manner.
>
> I prefer the approach taken by my diff-javatest tool
> of not caring about how many tests failed with the
> reference jdk. Just run them twice, and look for
> *new* failures.

What a pain... and what a loss, so many of these tests can become
healthy functioning members of society, given a little support and
a helping hand. And if they are just bad tests, I agree with
Mr. Scrooge let's 'decrease the surplus population'. ;^)

On diff-javatest, what if the moon is full, and the reference jdk
passed a test but your jdk didn't? For no good reason. I hate that.

>
>> Now it's time to go through the ProblemList.txt and do a little
>> triage, file a few bugs, fix what tests can be fixed, and/or
>> correct my ProblemList.txt file if I got it wrong.
>>
>>> BTW, I run regtests on java.util collections regularly,
>>> but only on Linux.
>> I think it is an expected situation that many developers can only
>> run the tests on one platform.
>>
>> But do you expect the gatekeeper to run all the tests on all the
>> platforms?
>> And if for example, your tests fail on Windows you would be
>> given X time units to fix it? Could you?
>
> Of course, you want the perfect combination of nasty and nice.
> For some test failures, it's unreasonable to expect the guilty party
> to fix them. But they should be prepared to be helpful.

Agreed, but both helpful and pro-active in preventing it from happening
again. We want "good citizen" developers. Which in general I think we have,
I'm just trying to make sure there are lots of tools and mechanisms to
allow everyone to be "good citizens". ;^)

-kto

>
> Martin

Re: Allow Java's ELF symtab reader to use separate debuginfo files

2009-12-09T10:10:12Z

Kelly O'Hair wrote:

> Andrew John Hughes wrote:
>> 2009/12/9 Andrew Haley <aph@...>:
>>> This is https://bugzilla.redhat.com/show_bug.cgi?id=541548
>>> The symptom is that jmap doesn't work because target libraries are stripped.
>>> The fix is to allow the symtab reader to use the separate debuginfo files
>>> that are available for all (AFAIK) GNU/Linux distributions.
>>>
>>> I'll commit this to the IcedTea repo, with a view to pushing it upstream
>>> to OpenJDK if it works on all distros.
>>>
>>> Andrew.
>>>
>> It would be good to see this upstream. The obvious compatibility
>> issue I can see is hardcoding /usr/lib/debug. Passing that in from
>> the Makefile, thus allowing to be overridden if necessary, might be
>> better.
>> i.e.
>>
>> static const char debug_file_directory[] = "/usr/lib/debug";
>>
>> becomes
>>
>> #ifndef DEBUG_FILE_DIRECTORY
>> #define DEBUG_FILE_DIRECTORY "/usr/lib/debug"
>> #endif
>>
>> with apppropriate logic in the OpenJDK makefile(s).
>>
>> Only other comment is that your ChangeLog should refer to openjdk, not
>> openjdk-ecj as we want this for all builds. The patch itself is
>> correct, it's just the ChangeLog.
>>
>> Redirecting this mail to the HotSpot and serviceability lists rather
>> than tl as it concerns the serviceability tools in HotSpot.
>
> Is there some other way than baking in 256 hex constants into our sources?

I'm sure it's perfectly possible to generate the CRC-32 tables at runtime.
However, the CRC function is defined in terms of this exact source code,
see http://sourceware.org/gdb/current/onlinedocs/gdb/Separate-Debug-Files.html#Separate-Debug-Files

> Why can't this logic be in a system include file somwehere?

Apart from "because it isn't", I can think of no good reason. Clearly
there should be a common API for this stuff, but it is spread around
various tools and libraries with licences we can't use, such as GPLv3+.

Andrew.

Re: Allow Java's ELF symtab reader to use separate debuginfo files

2009-12-09T09:30:19Z

2009/12/9 Andrew Haley <aph@...>:

> Andrew John Hughes wrote:
>> 2009/12/9 Andrew Haley <aph@...>:
>>> This is https://bugzilla.redhat.com/show_bug.cgi?id=541548
>>> The symptom is that jmap doesn't work because target libraries are stripped.
>>> The fix is to allow the symtab reader to use the separate debuginfo files
>>> that are available for all (AFAIK) GNU/Linux distributions.
>>>
>>> I'll commit this to the IcedTea repo, with a view to pushing it upstream
>>> to OpenJDK if it works on all distros.
>>
>> It would be good to see this upstream. The obvious compatibility
>> issue I can see is hardcoding /usr/lib/debug. Passing that in from
>> the Makefile, thus allowing to be overridden if necessary, might be
>> better.
>> i.e.
>>
>> static const char debug_file_directory[] = "/usr/lib/debug";
>>
>> becomes
>>
>> #ifndef DEBUG_FILE_DIRECTORY
>> #define DEBUG_FILE_DIRECTORY "/usr/lib/debug"
>> #endif
>>
>> with apppropriate logic in the OpenJDK makefile(s).
>
> I think this directory is generally assumed to be hard-coded everywhere. I
> will make such a change if this patch won't be committed otherwise, but I do
> not believe that it will ever be used.
>

Ah, ok. This was the only possibly distro-specific issue I could see
in the patch.

>> Only other comment is that your ChangeLog should refer to openjdk, not
>> openjdk-ecj as we want this for all builds. The patch itself is
>> correct, it's just the ChangeLog.
>
> Yeah, I noticed this already.
>
> Thanks,
> Andrew.
>

--
Andrew :-)

Free Java Software Engineer
Red Hat, Inc. (http://www.redhat.com)

Support Free Java!
Contribute to GNU Classpath and the OpenJDK
http://www.gnu.org/software/classpath
http://openjdk.java.net

PGP Key: 94EFD9D8 (http://subkeys.pgp.net)
Fingerprint: F8EF F1EA 401E 2E60 15FA 7927 142C 2591 94EF D9D8

Re: Allow Java's ELF symtab reader to use separate debuginfo files

2009-12-09T09:25:45Z

Andrew John Hughes wrote:

> 2009/12/9 Andrew Haley <aph@...>:
>> This is https://bugzilla.redhat.com/show_bug.cgi?id=541548
>> The symptom is that jmap doesn't work because target libraries are stripped.
>> The fix is to allow the symtab reader to use the separate debuginfo files
>> that are available for all (AFAIK) GNU/Linux distributions.
>>
>> I'll commit this to the IcedTea repo, with a view to pushing it upstream
>> to OpenJDK if it works on all distros.
>>
>> Andrew.
>>
>
> It would be good to see this upstream. The obvious compatibility
> issue I can see is hardcoding /usr/lib/debug. Passing that in from
> the Makefile, thus allowing to be overridden if necessary, might be
> better.
> i.e.
>
> static const char debug_file_directory[] = "/usr/lib/debug";
>
> becomes
>
> #ifndef DEBUG_FILE_DIRECTORY
> #define DEBUG_FILE_DIRECTORY "/usr/lib/debug"
> #endif
>
> with apppropriate logic in the OpenJDK makefile(s).
>
> Only other comment is that your ChangeLog should refer to openjdk, not
> openjdk-ecj as we want this for all builds. The patch itself is
> correct, it's just the ChangeLog.
>
> Redirecting this mail to the HotSpot and serviceability lists rather
> than tl as it concerns the serviceability tools in HotSpot.

Is there some other way than baking in 256 hex constants into our sources?

Why can't this logic be in a system include file somwehere?

-kto

>
>>
>> 2009-12-09 Andrew Haley <aph@...>
>>
>> * openjdk-ecj/hotspot/agent/src/os/linux/symtab.c (debug_file_directory):
>> New variable.
>> (gnu_debuglink_crc32): New.
>> (build_symtab_internal): Rename from build_symtab, make static.
>> Add logic to look inside debuginfo files.
>> (open_debug_file, find_section_by_name, open_file_from_debug_link,
>> build_id_to_debug_filename, build_symtab_from_build_id,
>> build_symtab): New.
>> * openjdk-ecj/hotspot/agent/src/os/linux/libproc_impl.c (add_lib_info_fd):
>> Pass FILENAME to build_symtab().
>> * openjdk-ecj/hotspot/agent/src/os/linux/symtab.h (build_symtab):
>> Add FILENAME arg.
>>
>> diff -u ../openjdk/icedtea6/openjdk/hotspot/agent/src/os/linux/libproc_impl.c openjdk/hotspot/agent/src/os/linux/libproc_impl.c
>> --- ../openjdk/icedtea6/openjdk/hotspot/agent/src/os/linux/libproc_impl.c 2009-05-15 00:36:38.000000000 +0100
>> +++ openjdk/hotspot/agent/src/os/linux/libproc_impl.c 2009-12-04 16:29:27.552837905 +0000
>> @@ -174,7 +174,7 @@
>> return NULL;
>> }
>>
>> - newlib->symtab = build_symtab(newlib->fd);
>> + newlib->symtab = build_symtab(newlib->fd, libname);
>> if (newlib->symtab == NULL) {
>> print_debug("symbol table build failed for %s\n", newlib->name);
>> }
>> diff -u ../openjdk/icedtea6/openjdk/hotspot/agent/src/os/linux/symtab.c openjdk/hotspot/agent/src/os/linux/symtab.c
>> --- ../openjdk/icedtea6/openjdk/hotspot/agent/src/os/linux/symtab.c 2009-05-15 00:36:38.000000000 +0100
>> +++ openjdk/hotspot/agent/src/os/linux/symtab.c 2009-12-07 13:53:13.107734342 +0000
>> @@ -53,8 +53,272 @@
>> struct hsearch_data *hash_table;
>> } symtab_t;
>>
>> -// read symbol table from given fd.
>> -struct symtab* build_symtab(int fd) {
>> +
>> +// Directory that contains global debuginfo files. In theory it
>> +// should be possible to change this, but in a Java environment there
>> +// is no obvious place to put a user interface to do it. Maybe this
>> +// could be set with an environment variable.
>> +static const char debug_file_directory[] = "/usr/lib/debug";
>> +
>> +/* The CRC used in gnu_debuglink, retrieved from
>> + http://sourceware.org/gdb/current/onlinedocs/gdb/Separate-Debug-Files.html#Separate-Debug-Files. */
>> +unsigned long gnu_debuglink_crc32 (unsigned long crc,
>> + unsigned char *buf, size_t len)
>> +{
>> + static const unsigned long crc32_table[256] =
>> + {
>> + 0x00000000, 0x77073096, 0xee0e612c, 0x990951ba, 0x076dc419,
>> + 0x706af48f, 0xe963a535, 0x9e6495a3, 0x0edb8832, 0x79dcb8a4,
>> + 0xe0d5e91e, 0x97d2d988, 0x09b64c2b, 0x7eb17cbd, 0xe7b82d07,
>> + 0x90bf1d91, 0x1db71064, 0x6ab020f2, 0xf3b97148, 0x84be41de,
>> + 0x1adad47d, 0x6ddde4eb, 0xf4d4b551, 0x83d385c7, 0x136c9856,
>> + 0x646ba8c0, 0xfd62f97a, 0x8a65c9ec, 0x14015c4f, 0x63066cd9,
>> + 0xfa0f3d63, 0x8d080df5, 0x3b6e20c8, 0x4c69105e, 0xd56041e4,
>> + 0xa2677172, 0x3c03e4d1, 0x4b04d447, 0xd20d85fd, 0xa50ab56b,
>> + 0x35b5a8fa, 0x42b2986c, 0xdbbbc9d6, 0xacbcf940, 0x32d86ce3,
>> + 0x45df5c75, 0xdcd60dcf, 0xabd13d59, 0x26d930ac, 0x51de003a,
>> + 0xc8d75180, 0xbfd06116, 0x21b4f4b5, 0x56b3c423, 0xcfba9599,
>> + 0xb8bda50f, 0x2802b89e, 0x5f058808, 0xc60cd9b2, 0xb10be924,
>> + 0x2f6f7c87, 0x58684c11, 0xc1611dab, 0xb6662d3d, 0x76dc4190,
>> + 0x01db7106, 0x98d220bc, 0xefd5102a, 0x71b18589, 0x06b6b51f,
>> + 0x9fbfe4a5, 0xe8b8d433, 0x7807c9a2, 0x0f00f934, 0x9609a88e,
>> + 0xe10e9818, 0x7f6a0dbb, 0x086d3d2d, 0x91646c97, 0xe6635c01,
>> + 0x6b6b51f4, 0x1c6c6162, 0x856530d8, 0xf262004e, 0x6c0695ed,
>> + 0x1b01a57b, 0x8208f4c1, 0xf50fc457, 0x65b0d9c6, 0x12b7e950,
>> + 0x8bbeb8ea, 0xfcb9887c, 0x62dd1ddf, 0x15da2d49, 0x8cd37cf3,
>> + 0xfbd44c65, 0x4db26158, 0x3ab551ce, 0xa3bc0074, 0xd4bb30e2,
>> + 0x4adfa541, 0x3dd895d7, 0xa4d1c46d, 0xd3d6f4fb, 0x4369e96a,
>> + 0x346ed9fc, 0xad678846, 0xda60b8d0, 0x44042d73, 0x33031de5,
>> + 0xaa0a4c5f, 0xdd0d7cc9, 0x5005713c, 0x270241aa, 0xbe0b1010,
>> + 0xc90c2086, 0x5768b525, 0x206f85b3, 0xb966d409, 0xce61e49f,
>> + 0x5edef90e, 0x29d9c998, 0xb0d09822, 0xc7d7a8b4, 0x59b33d17,
>> + 0x2eb40d81, 0xb7bd5c3b, 0xc0ba6cad, 0xedb88320, 0x9abfb3b6,
>> + 0x03b6e20c, 0x74b1d29a, 0xead54739, 0x9dd277af, 0x04db2615,
>> + 0x73dc1683, 0xe3630b12, 0x94643b84, 0x0d6d6a3e, 0x7a6a5aa8,
>> + 0xe40ecf0b, 0x9309ff9d, 0x0a00ae27, 0x7d079eb1, 0xf00f9344,
>> + 0x8708a3d2, 0x1e01f268, 0x6906c2fe, 0xf762575d, 0x806567cb,
>> + 0x196c3671, 0x6e6b06e7, 0xfed41b76, 0x89d32be0, 0x10da7a5a,
>> + 0x67dd4acc, 0xf9b9df6f, 0x8ebeeff9, 0x17b7be43, 0x60b08ed5,
>> + 0xd6d6a3e8, 0xa1d1937e, 0x38d8c2c4, 0x4fdff252, 0xd1bb67f1,
>> + 0xa6bc5767, 0x3fb506dd, 0x48b2364b, 0xd80d2bda, 0xaf0a1b4c,
>> + 0x36034af6, 0x41047a60, 0xdf60efc3, 0xa867df55, 0x316e8eef,
>> + 0x4669be79, 0xcb61b38c, 0xbc66831a, 0x256fd2a0, 0x5268e236,
>> + 0xcc0c7795, 0xbb0b4703, 0x220216b9, 0x5505262f, 0xc5ba3bbe,
>> + 0xb2bd0b28, 0x2bb45a92, 0x5cb36a04, 0xc2d7ffa7, 0xb5d0cf31,
>> + 0x2cd99e8b, 0x5bdeae1d, 0x9b64c2b0, 0xec63f226, 0x756aa39c,
>> + 0x026d930a, 0x9c0906a9, 0xeb0e363f, 0x72076785, 0x05005713,
>> + 0x95bf4a82, 0xe2b87a14, 0x7bb12bae, 0x0cb61b38, 0x92d28e9b,
>> + 0xe5d5be0d, 0x7cdcefb7, 0x0bdbdf21, 0x86d3d2d4, 0xf1d4e242,
>> + 0x68ddb3f8, 0x1fda836e, 0x81be16cd, 0xf6b9265b, 0x6fb077e1,
>> + 0x18b74777, 0x88085ae6, 0xff0f6a70, 0x66063bca, 0x11010b5c,
>> + 0x8f659eff, 0xf862ae69, 0x616bffd3, 0x166ccf45, 0xa00ae278,
>> + 0xd70dd2ee, 0x4e048354, 0x3903b3c2, 0xa7672661, 0xd06016f7,
>> + 0x4969474d, 0x3e6e77db, 0xaed16a4a, 0xd9d65adc, 0x40df0b66,
>> + 0x37d83bf0, 0xa9bcae53, 0xdebb9ec5, 0x47b2cf7f, 0x30b5ffe9,
>> + 0xbdbdf21c, 0xcabac28a, 0x53b39330, 0x24b4a3a6, 0xbad03605,
>> + 0xcdd70693, 0x54de5729, 0x23d967bf, 0xb3667a2e, 0xc4614ab8,
>> + 0x5d681b02, 0x2a6f2b94, 0xb40bbe37, 0xc30c8ea1, 0x5a05df1b,
>> + 0x2d02ef8d
>> + };
>> + unsigned char *end;
>> +
>> + crc = ~crc & 0xffffffff;
>> + for (end = buf + len; buf < end; ++buf)
>> + crc = crc32_table[(crc ^ *buf) & 0xff] ^ (crc >> 8);
>> + return ~crc & 0xffffffff;
>> +}
>> +
>> +/* Open a debuginfo file and check its CRC. If it exists and the CRC
>> + matches return its fd. */
>> +static int
>> +open_debug_file (const char *pathname, unsigned int crc)
>> +{
>> + unsigned int file_crc = 0;
>> + unsigned char buffer[8 * 1024];
>> +
>> + int fd = pathmap_open(pathname);
>> +
>> + if (fd < 0)
>> + return -1;
>> +
>> + lseek(fd, 0, SEEK_SET);
>> +
>> + for (;;) {
>> + int len = read(fd, buffer, sizeof buffer);
>> + if (len <= 0)
>> + break;
>> + file_crc = gnu_debuglink_crc32(file_crc, buffer, len);
>> + }
>> +
>> + if (crc == file_crc)
>> + return fd;
>> + else {
>> + close(fd);
>> + return -1;
>> + }
>> +}
>> +
>> +/* Find an ELF section. */
>> +static struct elf_section *find_section_by_name(char *name,
>> + int fd,
>> + ELF_EHDR *ehdr,
>> + ELF_SHDR *shbuf,
>> + struct elf_section *scn_cache)
>> +{
>> + ELF_SHDR* cursct = NULL;
>> + char *strtab;
>> + int cnt;
>> +
>> + if (scn_cache[ehdr->e_shstrndx].c_data == NULL) {
>> + if ((scn_cache[ehdr->e_shstrndx].c_data
>> + = read_section_data(fd, ehdr, cursct)) == NULL) {
>> + return NULL;
>> + }
>> + }
>> +
>> + strtab = scn_cache[ehdr->e_shstrndx].c_data;
>> +
>> + for (cursct = shbuf, cnt = 0;
>> + cnt < ehdr->e_shnum;
>> + cnt++, cursct++) {
>> + if (strcmp(cursct->sh_name + strtab, name) == 0) {
>> + scn_cache[cnt].c_data = read_section_data(fd, ehdr, cursct);
>> + return &scn_cache[cnt];
>> + }
>> + }
>> +
>> + return NULL;
>> +}
>> +
>> +/* Look for a ".gnu_debuglink" section. If one exists, try to open a
>> + suitable debuginfo file. */
>> +static int open_file_from_debug_link(const char *name,
>> + int fd,
>> + ELF_EHDR *ehdr,
>> + ELF_SHDR *shbuf,
>> + struct elf_section *scn_cache)
>> +{
>> + struct elf_section *debug_link = find_section_by_name(".gnu_debuglink", fd, ehdr,
>> + shbuf, scn_cache);
>> + int debug_fd;
>> + char *debug_filename = debug_link->c_data;
>> + int offset = strlen(debug_filename) + 3 >> 2;
>> + static unsigned int crc;
>> + crc = ((unsigned int*)debug_link->c_data)[offset];
>> + char *debug_pathname = malloc(strlen(debug_filename)
>> + + strlen(name)
>> + + strlen(".debug/")
>> + + strlen(debug_file_directory)
>> + + 2);
>> + strcpy(debug_pathname, name);
>> + char *last_slash = strrchr(debug_pathname, '/');
>> + if (last_slash == NULL)
>> + return -1;
>> +
>> + /* Look in the same directory as the object. */
>> + strcpy(last_slash+1, debug_filename);
>> +
>> + debug_fd = open_debug_file(debug_pathname, crc);
>> + if (debug_fd >= 0) {
>> + free(debug_pathname);
>> + return debug_fd;
>> + }
>> +
>> + /* Look in a subdirectory named ".debug". */
>> + strcpy(last_slash+1, ".debug/");
>> + strcat(last_slash, debug_filename);
>> +
>> + debug_fd = open_debug_file(debug_pathname, crc);
>> + if (debug_fd >= 0) {
>> + free(debug_pathname);
>> + return debug_fd;
>> + }
>> +
>> + /* Look in /usr/lib/debug + the full pathname. */
>> + strcpy(debug_pathname, debug_file_directory);
>> + strcat(debug_pathname, name);
>> + last_slash = strrchr(debug_pathname, '/');
>> + strcpy(last_slash+1, debug_filename);
>> +
>> + debug_fd = open_debug_file(debug_pathname, crc);
>> + if (debug_fd >= 0) {
>> + free(debug_pathname);
>> + return debug_fd;
>> + }
>> +
>> + free(debug_pathname);
>> + return -1;
>> +}
>> +
>> +static struct symtab* build_symtab_internal(int fd, const char *filename, bool try_debuginfo);
>> +
>> +/* Look for a ".gnu_debuglink" section. If one exists, try to open a
>> + suitable debuginfo file and read a symbol table from it. */
>> +static struct symtab *build_symtab_from_debug_link(const char *name,
>> + int fd,
>> + ELF_EHDR *ehdr,
>> + ELF_SHDR *shbuf,
>> + struct elf_section *scn_cache)
>> +{
>> + fd = open_file_from_debug_link(name, fd, ehdr, shbuf, scn_cache);
>> +
>> + if (fd >= 0) {
>> + struct symtab *symtab = build_symtab_internal(fd, NULL, /* try_debuginfo */ false);
>> + close(fd);
>> + return symtab;
>> + }
>> +
>> + return NULL;
>> +}
>> +
>> +// Given a build_id, find the associated debuginfo file
>> +static char *
>> +build_id_to_debug_filename (size_t size, unsigned char *data)
>> +{
>> + char *filename, *s;
>> +
>> + filename = malloc(strlen (debug_file_directory) + (sizeof "/.build-id/" - 1) + 1
>> + + 2 * size + (sizeof ".debug" - 1) + 1);
>> + s = filename + sprintf (filename, "%s/.build-id/", debug_file_directory);
>> + if (size > 0)
>> + {
>> + size--;
>> + s += sprintf (s, "%02x", *data++);
>> + }
>> + if (size > 0)
>> + *s++ = '/';
>> + while (size-- > 0)
>> + s += sprintf (s, "%02x", *data++);
>> + strcpy (s, ".debug");
>> +
>> + return filename;
>> +}
>> +
>> +// Read a build ID note. Try to open any associated debuginfo file
>> +// and return its symtab
>> +static struct symtab* build_symtab_from_build_id(Elf64_Nhdr *note)
>> +{
>> + int fd;
>> + struct symtab *symtab = NULL;
>> +
>> + unsigned char *bytes
>> + = (unsigned char*)(note+1) + note->n_namesz;
>> + unsigned char *filename
>> + = (build_id_to_debug_filename (note->n_descsz, bytes));
>> +
>> + fd = pathmap_open(filename);
>> + if (fd >= 0) {
>> + symtab = build_symtab_internal(fd, NULL, /* try_debuginfo */ false);
>> + close(fd);
>> + }
>> + free(filename);
>> +
>> + return symtab;
>> +}
>> +
>> +// read symbol table from given fd. If try_debuginfo) is true, also
>> +// try to open an associated debuginfo file
>> +static struct symtab* build_symtab_internal(int fd, const char *filename, bool try_debuginfo) {
>> ELF_EHDR ehdr;
>> char *names = NULL;
>> struct symtab* symtab = NULL;
>> @@ -90,7 +354,8 @@
>>
>> for (cursct = shbuf, cnt = 0; cnt < ehdr.e_shnum; cnt++) {
>> scn_cache[cnt].c_shdr = cursct;
>> - if (cursct->sh_type == SHT_SYMTAB || cursct->sh_type == SHT_STRTAB) {
>> + if (cursct->sh_type == SHT_SYMTAB || cursct->sh_type == SHT_STRTAB
>> + || cursct->sh_type == SHT_NOTE) {
>> if ( (scn_cache[cnt].c_data = read_section_data(fd, &ehdr, cursct)) == NULL) {
>> goto quit;
>> }
>> @@ -163,6 +428,28 @@
>> }
>> }
>>
>> + // If we have no symtab in this file, look for a separate debuginfo file
>> + if (try_debuginfo) {
>> + // First we look for a Build ID
>> + for (cursct = shbuf, cnt = 0;
>> + symtab == NULL && cnt < ehdr.e_shnum;
>> + cnt++) {
>> + if (cursct->sh_type == SHT_NOTE) {
>> + Elf64_Nhdr *note = (Elf64_Nhdr *)scn_cache[cnt].c_data;
>> + if (note->n_type == NT_GNU_BUILD_ID) {
>> + symtab = build_symtab_from_build_id(note);
>> + }
>> + }
>> + cursct++;
>> + }
>> +
>> + // Then, if that doesn't work, the debug link
>> + if (symtab == NULL) {
>> + symtab = build_symtab_from_debug_link(filename, fd, &ehdr, shbuf,
>> + scn_cache);
>> + }
>> + }
>> +
>> quit:
>> if (shbuf) free(shbuf);
>> if (phbuf) free(phbuf);
>> @@ -177,6 +464,11 @@
>> return symtab;
>> }
>>
>> +struct symtab* build_symtab(int fd, const char *filename) {
>> + return build_symtab_internal(fd, filename, /* try_debuginfo */ true);
>> +}
>> +
>> +
>> void destroy_symtab(struct symtab* symtab) {
>> if (!symtab) return;
>> if (symtab->strs) free(symtab->strs);
>> diff -u ../openjdk/icedtea6/openjdk/hotspot/agent/src/os/linux/symtab.h openjdk/hotspot/agent/src/os/linux/symtab.h
>> --- ../openjdk/icedtea6/openjdk/hotspot/agent/src/os/linux/symtab.h 2009-05-15 00:36:38.000000000 +0100
>> +++ openjdk/hotspot/agent/src/os/linux/symtab.h 2009-12-04 16:30:38.797838015 +0000
>> @@ -32,7 +32,7 @@
>> struct symtab;
>>
>> // build symbol table for a given ELF file descriptor
>> -struct symtab* build_symtab(int fd);
>> +struct symtab* build_symtab(int fd, const char *filename);
>>
>> // destroy the symbol table
>> void destroy_symtab(struct symtab* symtab);
>>
>>
>>
>
>
>

Re: Allow Java's ELF symtab reader to use separate debuginfo files

2009-12-09T08:51:00Z

Andrew John Hughes wrote:

> 2009/12/9 Andrew Haley <aph@...>:
>> This is https://bugzilla.redhat.com/show_bug.cgi?id=541548
>> The symptom is that jmap doesn't work because target libraries are stripped.
>> The fix is to allow the symtab reader to use the separate debuginfo files
>> that are available for all (AFAIK) GNU/Linux distributions.
>>
>> I'll commit this to the IcedTea repo, with a view to pushing it upstream
>> to OpenJDK if it works on all distros.
>
> It would be good to see this upstream. The obvious compatibility
> issue I can see is hardcoding /usr/lib/debug. Passing that in from
> the Makefile, thus allowing to be overridden if necessary, might be
> better.
> i.e.
>
> static const char debug_file_directory[] = "/usr/lib/debug";
>
> becomes
>
> #ifndef DEBUG_FILE_DIRECTORY
> #define DEBUG_FILE_DIRECTORY "/usr/lib/debug"
> #endif
>
> with apppropriate logic in the OpenJDK makefile(s).

I think this directory is generally assumed to be hard-coded everywhere. I
will make such a change if this patch won't be committed otherwise, but I do
not believe that it will ever be used.

> Only other comment is that your ChangeLog should refer to openjdk, not
> openjdk-ecj as we want this for all builds. The patch itself is
> correct, it's just the ChangeLog.

Yeah, I noticed this already.

Thanks,
Andrew.

Re: Allow Java's ELF symtab reader to use separate debuginfo files

2009-12-09T08:26:42Z

2009/12/9 Andrew Haley <aph@...>:

> This is https://bugzilla.redhat.com/show_bug.cgi?id=541548
> The symptom is that jmap doesn't work because target libraries are stripped.
> The fix is to allow the symtab reader to use the separate debuginfo files
> that are available for all (AFAIK) GNU/Linux distributions.
>
> I'll commit this to the IcedTea repo, with a view to pushing it upstream
> to OpenJDK if it works on all distros.
>
> Andrew.
>

It would be good to see this upstream. The obvious compatibility
issue I can see is hardcoding /usr/lib/debug. Passing that in from
the Makefile, thus allowing to be overridden if necessary, might be
better.
i.e.

static const char debug_file_directory[] = "/usr/lib/debug";

becomes

#ifndef DEBUG_FILE_DIRECTORY
#define DEBUG_FILE_DIRECTORY "/usr/lib/debug"
#endif

with apppropriate logic in the OpenJDK makefile(s).

Only other comment is that your ChangeLog should refer to openjdk, not
openjdk-ecj as we want this for all builds. The patch itself is
correct, it's just the ChangeLog.

Redirecting this mail to the HotSpot and serviceability lists rather
than tl as it concerns the serviceability tools in HotSpot.

>
>
> 2009-12-09 Andrew Haley <aph@...>
>
> * openjdk-ecj/hotspot/agent/src/os/linux/symtab.c (debug_file_directory):
> New variable.
> (gnu_debuglink_crc32): New.
> (build_symtab_internal): Rename from build_symtab, make static.
> Add logic to look inside debuginfo files.
> (open_debug_file, find_section_by_name, open_file_from_debug_link,
> build_id_to_debug_filename, build_symtab_from_build_id,
> build_symtab): New.
> * openjdk-ecj/hotspot/agent/src/os/linux/libproc_impl.c (add_lib_info_fd):
> Pass FILENAME to build_symtab().
> * openjdk-ecj/hotspot/agent/src/os/linux/symtab.h (build_symtab):
> Add FILENAME arg.
>
> diff -u ../openjdk/icedtea6/openjdk/hotspot/agent/src/os/linux/libproc_impl.c openjdk/hotspot/agent/src/os/linux/libproc_impl.c
> --- ../openjdk/icedtea6/openjdk/hotspot/agent/src/os/linux/libproc_impl.c 2009-05-15 00:36:38.000000000 +0100
> +++ openjdk/hotspot/agent/src/os/linux/libproc_impl.c 2009-12-04 16:29:27.552837905 +0000
> @@ -174,7 +174,7 @@
> return NULL;
> }
>
> - newlib->symtab = build_symtab(newlib->fd);
> + newlib->symtab = build_symtab(newlib->fd, libname);
> if (newlib->symtab == NULL) {
> print_debug("symbol table build failed for %s\n", newlib->name);
> }
> diff -u ../openjdk/icedtea6/openjdk/hotspot/agent/src/os/linux/symtab.c openjdk/hotspot/agent/src/os/linux/symtab.c
> --- ../openjdk/icedtea6/openjdk/hotspot/agent/src/os/linux/symtab.c 2009-05-15 00:36:38.000000000 +0100
> +++ openjdk/hotspot/agent/src/os/linux/symtab.c 2009-12-07 13:53:13.107734342 +0000
> @@ -53,8 +53,272 @@
> struct hsearch_data *hash_table;
> } symtab_t;
>
> -// read symbol table from given fd.
> -struct symtab* build_symtab(int fd) {
> +
> +// Directory that contains global debuginfo files. In theory it
> +// should be possible to change this, but in a Java environment there
> +// is no obvious place to put a user interface to do it. Maybe this
> +// could be set with an environment variable.
> +static const char debug_file_directory[] = "/usr/lib/debug";
> +
> +/* The CRC used in gnu_debuglink, retrieved from
> + http://sourceware.org/gdb/current/onlinedocs/gdb/Separate-Debug-Files.html#Separate-Debug-Files. */
> +unsigned long gnu_debuglink_crc32 (unsigned long crc,
> + unsigned char *buf, size_t len)
> +{
> + static const unsigned long crc32_table[256] =
> + {
> + 0x00000000, 0x77073096, 0xee0e612c, 0x990951ba, 0x076dc419,
> + 0x706af48f, 0xe963a535, 0x9e6495a3, 0x0edb8832, 0x79dcb8a4,
> + 0xe0d5e91e, 0x97d2d988, 0x09b64c2b, 0x7eb17cbd, 0xe7b82d07,
> + 0x90bf1d91, 0x1db71064, 0x6ab020f2, 0xf3b97148, 0x84be41de,
> + 0x1adad47d, 0x6ddde4eb, 0xf4d4b551, 0x83d385c7, 0x136c9856,
> + 0x646ba8c0, 0xfd62f97a, 0x8a65c9ec, 0x14015c4f, 0x63066cd9,
> + 0xfa0f3d63, 0x8d080df5, 0x3b6e20c8, 0x4c69105e, 0xd56041e4,
> + 0xa2677172, 0x3c03e4d1, 0x4b04d447, 0xd20d85fd, 0xa50ab56b,
> + 0x35b5a8fa, 0x42b2986c, 0xdbbbc9d6, 0xacbcf940, 0x32d86ce3,
> + 0x45df5c75, 0xdcd60dcf, 0xabd13d59, 0x26d930ac, 0x51de003a,
> + 0xc8d75180, 0xbfd06116, 0x21b4f4b5, 0x56b3c423, 0xcfba9599,
> + 0xb8bda50f, 0x2802b89e, 0x5f058808, 0xc60cd9b2, 0xb10be924,
> + 0x2f6f7c87, 0x58684c11, 0xc1611dab, 0xb6662d3d, 0x76dc4190,
> + 0x01db7106, 0x98d220bc, 0xefd5102a, 0x71b18589, 0x06b6b51f,
> + 0x9fbfe4a5, 0xe8b8d433, 0x7807c9a2, 0x0f00f934, 0x9609a88e,
> + 0xe10e9818, 0x7f6a0dbb, 0x086d3d2d, 0x91646c97, 0xe6635c01,
> + 0x6b6b51f4, 0x1c6c6162, 0x856530d8, 0xf262004e, 0x6c0695ed,
> + 0x1b01a57b, 0x8208f4c1, 0xf50fc457, 0x65b0d9c6, 0x12b7e950,
> + 0x8bbeb8ea, 0xfcb9887c, 0x62dd1ddf, 0x15da2d49, 0x8cd37cf3,
> + 0xfbd44c65, 0x4db26158, 0x3ab551ce, 0xa3bc0074, 0xd4bb30e2,
> + 0x4adfa541, 0x3dd895d7, 0xa4d1c46d, 0xd3d6f4fb, 0x4369e96a,
> + 0x346ed9fc, 0xad678846, 0xda60b8d0, 0x44042d73, 0x33031de5,
> + 0xaa0a4c5f, 0xdd0d7cc9, 0x5005713c, 0x270241aa, 0xbe0b1010,
> + 0xc90c2086, 0x5768b525, 0x206f85b3, 0xb966d409, 0xce61e49f,
> + 0x5edef90e, 0x29d9c998, 0xb0d09822, 0xc7d7a8b4, 0x59b33d17,
> + 0x2eb40d81, 0xb7bd5c3b, 0xc0ba6cad, 0xedb88320, 0x9abfb3b6,
> + 0x03b6e20c, 0x74b1d29a, 0xead54739, 0x9dd277af, 0x04db2615,
> + 0x73dc1683, 0xe3630b12, 0x94643b84, 0x0d6d6a3e, 0x7a6a5aa8,
> + 0xe40ecf0b, 0x9309ff9d, 0x0a00ae27, 0x7d079eb1, 0xf00f9344,
> + 0x8708a3d2, 0x1e01f268, 0x6906c2fe, 0xf762575d, 0x806567cb,
> + 0x196c3671, 0x6e6b06e7, 0xfed41b76, 0x89d32be0, 0x10da7a5a,
> + 0x67dd4acc, 0xf9b9df6f, 0x8ebeeff9, 0x17b7be43, 0x60b08ed5,
> + 0xd6d6a3e8, 0xa1d1937e, 0x38d8c2c4, 0x4fdff252, 0xd1bb67f1,
> + 0xa6bc5767, 0x3fb506dd, 0x48b2364b, 0xd80d2bda, 0xaf0a1b4c,
> + 0x36034af6, 0x41047a60, 0xdf60efc3, 0xa867df55, 0x316e8eef,
> + 0x4669be79, 0xcb61b38c, 0xbc66831a, 0x256fd2a0, 0x5268e236,
> + 0xcc0c7795, 0xbb0b4703, 0x220216b9, 0x5505262f, 0xc5ba3bbe,
> + 0xb2bd0b28, 0x2bb45a92, 0x5cb36a04, 0xc2d7ffa7, 0xb5d0cf31,
> + 0x2cd99e8b, 0x5bdeae1d, 0x9b64c2b0, 0xec63f226, 0x756aa39c,
> + 0x026d930a, 0x9c0906a9, 0xeb0e363f, 0x72076785, 0x05005713,
> + 0x95bf4a82, 0xe2b87a14, 0x7bb12bae, 0x0cb61b38, 0x92d28e9b,
> + 0xe5d5be0d, 0x7cdcefb7, 0x0bdbdf21, 0x86d3d2d4, 0xf1d4e242,
> + 0x68ddb3f8, 0x1fda836e, 0x81be16cd, 0xf6b9265b, 0x6fb077e1,
> + 0x18b74777, 0x88085ae6, 0xff0f6a70, 0x66063bca, 0x11010b5c,
> + 0x8f659eff, 0xf862ae69, 0x616bffd3, 0x166ccf45, 0xa00ae278,
> + 0xd70dd2ee, 0x4e048354, 0x3903b3c2, 0xa7672661, 0xd06016f7,
> + 0x4969474d, 0x3e6e77db, 0xaed16a4a, 0xd9d65adc, 0x40df0b66,
> + 0x37d83bf0, 0xa9bcae53, 0xdebb9ec5, 0x47b2cf7f, 0x30b5ffe9,
> + 0xbdbdf21c, 0xcabac28a, 0x53b39330, 0x24b4a3a6, 0xbad03605,
> + 0xcdd70693, 0x54de5729, 0x23d967bf, 0xb3667a2e, 0xc4614ab8,
> + 0x5d681b02, 0x2a6f2b94, 0xb40bbe37, 0xc30c8ea1, 0x5a05df1b,
> + 0x2d02ef8d
> + };
> + unsigned char *end;
> +
> + crc = ~crc & 0xffffffff;
> + for (end = buf + len; buf < end; ++buf)
> + crc = crc32_table[(crc ^ *buf) & 0xff] ^ (crc >> 8);
> + return ~crc & 0xffffffff;
> +}
> +
> +/* Open a debuginfo file and check its CRC. If it exists and the CRC
> + matches return its fd. */
> +static int
> +open_debug_file (const char *pathname, unsigned int crc)
> +{
> + unsigned int file_crc = 0;
> + unsigned char buffer[8 * 1024];
> +
> + int fd = pathmap_open(pathname);
> +
> + if (fd < 0)
> + return -1;
> +
> + lseek(fd, 0, SEEK_SET);
> +
> + for (;;) {
> + int len = read(fd, buffer, sizeof buffer);
> + if (len <= 0)
> + break;
> + file_crc = gnu_debuglink_crc32(file_crc, buffer, len);
> + }
> +
> + if (crc == file_crc)
> + return fd;
> + else {
> + close(fd);
> + return -1;
> + }
> +}
> +
> +/* Find an ELF section. */
> +static struct elf_section *find_section_by_name(char *name,
> + int fd,
> + ELF_EHDR *ehdr,
> + ELF_SHDR *shbuf,
> + struct elf_section *scn_cache)
> +{
> + ELF_SHDR* cursct = NULL;
> + char *strtab;
> + int cnt;
> +
> + if (scn_cache[ehdr->e_shstrndx].c_data == NULL) {
> + if ((scn_cache[ehdr->e_shstrndx].c_data
> + = read_section_data(fd, ehdr, cursct)) == NULL) {
> + return NULL;
> + }
> + }
> +
> + strtab = scn_cache[ehdr->e_shstrndx].c_data;
> +
> + for (cursct = shbuf, cnt = 0;
> + cnt < ehdr->e_shnum;
> + cnt++, cursct++) {
> + if (strcmp(cursct->sh_name + strtab, name) == 0) {
> + scn_cache[cnt].c_data = read_section_data(fd, ehdr, cursct);
> + return &scn_cache[cnt];
> + }
> + }
> +
> + return NULL;
> +}
> +
> +/* Look for a ".gnu_debuglink" section. If one exists, try to open a
> + suitable debuginfo file. */
> +static int open_file_from_debug_link(const char *name,
> + int fd,
> + ELF_EHDR *ehdr,
> + ELF_SHDR *shbuf,
> + struct elf_section *scn_cache)
> +{
> + struct elf_section *debug_link = find_section_by_name(".gnu_debuglink", fd, ehdr,
> + shbuf, scn_cache);
> + int debug_fd;
> + char *debug_filename = debug_link->c_data;
> + int offset = strlen(debug_filename) + 3 >> 2;
> + static unsigned int crc;
> + crc = ((unsigned int*)debug_link->c_data)[offset];
> + char *debug_pathname = malloc(strlen(debug_filename)
> + + strlen(name)
> + + strlen(".debug/")
> + + strlen(debug_file_directory)
> + + 2);
> + strcpy(debug_pathname, name);
> + char *last_slash = strrchr(debug_pathname, '/');
> + if (last_slash == NULL)
> + return -1;
> +
> + /* Look in the same directory as the object. */
> + strcpy(last_slash+1, debug_filename);
> +
> + debug_fd = open_debug_file(debug_pathname, crc);
> + if (debug_fd >= 0) {
> + free(debug_pathname);
> + return debug_fd;
> + }
> +
> + /* Look in a subdirectory named ".debug". */
> + strcpy(last_slash+1, ".debug/");
> + strcat(last_slash, debug_filename);
> +
> + debug_fd = open_debug_file(debug_pathname, crc);
> + if (debug_fd >= 0) {
> + free(debug_pathname);
> + return debug_fd;
> + }
> +
> + /* Look in /usr/lib/debug + the full pathname. */
> + strcpy(debug_pathname, debug_file_directory);
> + strcat(debug_pathname, name);
> + last_slash = strrchr(debug_pathname, '/');
> + strcpy(last_slash+1, debug_filename);
> +
> + debug_fd = open_debug_file(debug_pathname, crc);
> + if (debug_fd >= 0) {
> + free(debug_pathname);
> + return debug_fd;
> + }
> +
> + free(debug_pathname);
> + return -1;
> +}
> +
> +static struct symtab* build_symtab_internal(int fd, const char *filename, bool try_debuginfo);
> +
> +/* Look for a ".gnu_debuglink" section. If one exists, try to open a
> + suitable debuginfo file and read a symbol table from it. */
> +static struct symtab *build_symtab_from_debug_link(const char *name,
> + int fd,
> + ELF_EHDR *ehdr,
> + ELF_SHDR *shbuf,
> + struct elf_section *scn_cache)
> +{
> + fd = open_file_from_debug_link(name, fd, ehdr, shbuf, scn_cache);
> +
> + if (fd >= 0) {
> + struct symtab *symtab = build_symtab_internal(fd, NULL, /* try_debuginfo */ false);
> + close(fd);
> + return symtab;
> + }
> +
> + return NULL;
> +}
> +
> +// Given a build_id, find the associated debuginfo file
> +static char *
> +build_id_to_debug_filename (size_t size, unsigned char *data)
> +{
> + char *filename, *s;
> +
> + filename = malloc(strlen (debug_file_directory) + (sizeof "/.build-id/" - 1) + 1
> + + 2 * size + (sizeof ".debug" - 1) + 1);
> + s = filename + sprintf (filename, "%s/.build-id/", debug_file_directory);
> + if (size > 0)
> + {
> + size--;
> + s += sprintf (s, "%02x", *data++);
> + }
> + if (size > 0)
> + *s++ = '/';
> + while (size-- > 0)
> + s += sprintf (s, "%02x", *data++);
> + strcpy (s, ".debug");
> +
> + return filename;
> +}
> +
> +// Read a build ID note. Try to open any associated debuginfo file
> +// and return its symtab
> +static struct symtab* build_symtab_from_build_id(Elf64_Nhdr *note)
> +{
> + int fd;
> + struct symtab *symtab = NULL;
> +
> + unsigned char *bytes
> + = (unsigned char*)(note+1) + note->n_namesz;
> + unsigned char *filename
> + = (build_id_to_debug_filename (note->n_descsz, bytes));
> +
> + fd = pathmap_open(filename);
> + if (fd >= 0) {
> + symtab = build_symtab_internal(fd, NULL, /* try_debuginfo */ false);
> + close(fd);
> + }
> + free(filename);
> +
> + return symtab;
> +}
> +
> +// read symbol table from given fd. If try_debuginfo) is true, also
> +// try to open an associated debuginfo file
> +static struct symtab* build_symtab_internal(int fd, const char *filename, bool try_debuginfo) {
> ELF_EHDR ehdr;
> char *names = NULL;
> struct symtab* symtab = NULL;
> @@ -90,7 +354,8 @@
>
> for (cursct = shbuf, cnt = 0; cnt < ehdr.e_shnum; cnt++) {
> scn_cache[cnt].c_shdr = cursct;
> - if (cursct->sh_type == SHT_SYMTAB || cursct->sh_type == SHT_STRTAB) {
> + if (cursct->sh_type == SHT_SYMTAB || cursct->sh_type == SHT_STRTAB
> + || cursct->sh_type == SHT_NOTE) {
> if ( (scn_cache[cnt].c_data = read_section_data(fd, &ehdr, cursct)) == NULL) {
> goto quit;
> }
> @@ -163,6 +428,28 @@
> }
> }
>
> + // If we have no symtab in this file, look for a separate debuginfo file
> + if (try_debuginfo) {
> + // First we look for a Build ID
> + for (cursct = shbuf, cnt = 0;
> + symtab == NULL && cnt < ehdr.e_shnum;
> + cnt++) {
> + if (cursct->sh_type == SHT_NOTE) {
> + Elf64_Nhdr *note = (Elf64_Nhdr *)scn_cache[cnt].c_data;
> + if (note->n_type == NT_GNU_BUILD_ID) {
> + symtab = build_symtab_from_build_id(note);
> + }
> + }
> + cursct++;
> + }
> +
> + // Then, if that doesn't work, the debug link
> + if (symtab == NULL) {
> + symtab = build_symtab_from_debug_link(filename, fd, &ehdr, shbuf,
> + scn_cache);
> + }
> + }
> +
> quit:
> if (shbuf) free(shbuf);
> if (phbuf) free(phbuf);
> @@ -177,6 +464,11 @@
> return symtab;
> }
>
> +struct symtab* build_symtab(int fd, const char *filename) {
> + return build_symtab_internal(fd, filename, /* try_debuginfo */ true);
> +}
> +
> +
> void destroy_symtab(struct symtab* symtab) {
> if (!symtab) return;
> if (symtab->strs) free(symtab->strs);
> diff -u ../openjdk/icedtea6/openjdk/hotspot/agent/src/os/linux/symtab.h openjdk/hotspot/agent/src/os/linux/symtab.h
> --- ../openjdk/icedtea6/openjdk/hotspot/agent/src/os/linux/symtab.h 2009-05-15 00:36:38.000000000 +0100
> +++ openjdk/hotspot/agent/src/os/linux/symtab.h 2009-12-04 16:30:38.797838015 +0000
> @@ -32,7 +32,7 @@
> struct symtab;
>
> // build symbol table for a given ELF file descriptor
> -struct symtab* build_symtab(int fd);
> +struct symtab* build_symtab(int fd, const char *filename);
>
> // destroy the symbol table
> void destroy_symtab(struct symtab* symtab);
>
>
>

--
Andrew :-)

Free Java Software Engineer
Red Hat, Inc. (http://www.redhat.com)

Support Free Java!
Contribute to GNU Classpath and the OpenJDK
http://www.gnu.org/software/classpath
http://openjdk.java.net

PGP Key: 94EFD9D8 (http://subkeys.pgp.net)
Fingerprint: F8EF F1EA 401E 2E60 15FA 7927 142C 2591 94EF D9D8

hg: jdk7/tl/jdk: 6908628: ObjectIdentifier s11n test fails

2009-12-08T19:16:09Z

Changeset: db5c77621c6b
Author: weijun
Date: 2009-12-09 11:15 +0800
URL: http://hg.openjdk.java.net/jdk7/tl/jdk/rev/db5c77621c6b

6908628: ObjectIdentifier s11n test fails
Reviewed-by: xuelei

! test/sun/security/util/Oid/S11N.sh

Re: Need reviewers, jdk7 testing changes

2009-12-08T17:38:43Z

On Tue, Dec 8, 2009 at 16:42, Kelly O'Hair <Kelly.Ohair@...> wrote:

>> In theory, the process is supposed to prevent
>> regression test failures from ever creeping in - that's the whole point.
>> When they do (inevitably) creep in,
>> they are supposed to be aggressively targeted.
>> A gatekeeper demonstrates the failure to a developer,
>> and the developer is given X time units to fix the test breakage,
>> or face reversion of the breakage-inducing change.
>
> But it's not that simple. In some cases the cause of the failure is
> a change made to a different repository, by a different team,

Changes that get in this way are signs of process failure.
E.g. failing tests in java.util caused by hotspot commits
are signs the hotspot commit was inadequately tested.

> or even to the base system software via an update.
> The ProblemList.txt file was meant to deal with that situation,
> and also tests that are spuriously failing for unknown reasons.

I agree that flaky and platform-dependent failures are a big problem.

> When the gatekeeper can do what you say, that is great, and ideal.
> But I just don't see it happening that way in all situations.
>
>> I would like to see more effort devoted to fixing the tests
>> (or the code!) rather than adding infrastructure that might
>> have the effect of hiding the test failures.
>
> Sigh... I'm not trying to hide the failures, and if you haven't
> noticed, I have fixed quite a few tests myself.

I have noticed, and I do appreciate it.

I do some flaky test fixing myself.

> If anything, I'm making the fact that we have test failures more
> public, and the ultimate goal is to fix the tests.
> But I needed a baseline, a line in the sand, an expectation
> of what tests should always pass. And an ability to run all
> the tests in a timely manner.

I prefer the approach taken by my diff-javatest tool
of not caring about how many tests failed with the
reference jdk. Just run them twice, and look for
*new* failures.

> Now it's time to go through the ProblemList.txt and do a little
> triage, file a few bugs, fix what tests can be fixed, and/or
> correct my ProblemList.txt file if I got it wrong.
>
>>
>> BTW, I run regtests on java.util collections regularly,
>> but only on Linux.
>
> I think it is an expected situation that many developers can only
> run the tests on one platform.
>
> But do you expect the gatekeeper to run all the tests on all the
> platforms?
> And if for example, your tests fail on Windows you would be
> given X time units to fix it? Could you?

Of course, you want the perfect combination of nasty and nice.
For some test failures, it's unreasonable to expect the guilty party
to fix them. But they should be prepared to be helpful.

Martin

Re: Need reviewers, jdk7 testing changes

2009-12-08T16:42:32Z

Martin Buchholz wrote:
> On Mon, Dec 7, 2009 at 23:30, Martin Buchholz <martinrb@...> wrote:
>
>> You cannot exclude the Mother Of All Tests.
>> This is non-negotiable.
>
> More seriously...
> It's very difficult to manage creeping failures in regression tests.

Tell me about it. :^(

>
> In theory, the process is supposed to prevent
> regression test failures from ever creeping in - that's the whole point.
> When they do (inevitably) creep in,
> they are supposed to be aggressively targeted.
> A gatekeeper demonstrates the failure to a developer,
> and the developer is given X time units to fix the test breakage,
> or face reversion of the breakage-inducing change.

But it's not that simple. In some cases the cause of the failure is
a change made to a different repository, by a different team,
or even to the base system software via an update.
The ProblemList.txt file was meant to deal with that situation,
and also tests that are spuriously failing for unknown reasons.

When the gatekeeper can do what you say, that is great, and ideal.
But I just don't see it happening that way in all situations.

> I would like to see more effort devoted to fixing the tests
> (or the code!) rather than adding infrastructure that might
> have the effect of hiding the test failures.

Sigh... I'm not trying to hide the failures, and if you haven't
noticed, I have fixed quite a few tests myself.
If anything, I'm making the fact that we have test failures more
public, and the ultimate goal is to fix the tests.
But I needed a baseline, a line in the sand, an expectation
of what tests should always pass. And an ability to run all
the tests in a timely manner.

Now it's time to go through the ProblemList.txt and do a little
triage, file a few bugs, fix what tests can be fixed, and/or
correct my ProblemList.txt file if I got it wrong.

>
> BTW, I run regtests on java.util collections regularly,
> but only on Linux.

I think it is an expected situation that many developers can only
run the tests on one platform.

But do you expect the gatekeeper to run all the tests on all the
platforms?
And if for example, your tests fail on Windows you would be
given X time units to fix it? Could you?

-kto

>
> Martin

Re: Need reviewers, jdk7 testing changes

2009-12-08T15:35:29Z

On Mon, Dec 7, 2009 at 23:30, Martin Buchholz <martinrb@...> wrote:

> You cannot exclude the Mother Of All Tests.
> This is non-negotiable.

More seriously...
It's very difficult to manage creeping failures in regression tests.

In theory, the process is supposed to prevent
regression test failures from ever creeping in - that's the whole point.
When they do (inevitably) creep in,
they are supposed to be aggressively targeted.
A gatekeeper demonstrates the failure to a developer,
and the developer is given X time units to fix the test breakage,
or face reversion of the breakage-inducing change.
I would like to see more effort devoted to fixing the tests
(or the code!) rather than adding infrastructure that might
have the effect of hiding the test failures.

BTW, I run regtests on java.util collections regularly,
but only on Linux.

Martin

Re: Need reviewers, jdk7 testing changes

2009-12-08T13:41:17Z

Kelly O'Hair wrote:

>
> Need reviewer for JDK7 testing changes, e.g. testing via:
> cd jdk/make && gmake all images && cd ../test && gmake jdk_all
>
> 6906210: Fix another minor typo in test/Makefile
> http://cr.openjdk.java.net/~ohair/openjdk7/jdk7-tl-test-changes4/webrev/
>
> Dan had pointed out my typo in the jdk/test/Makefile ($->$$)
> but when I started running and re-running the tests I ran into
> more unstable tests. This webrev includes fixes to 3 tests
> and unfortuntately more additions to the ProblemList file.
>
> Tests added to the ProblemList:

A few comments on problem list membership:

366 # Need to be marked othervm, or changed to be samevm safe
367 java/lang/annotation/ParameterAnnotations.java
generic-all

This test uses a security manager.

499 # Problems with rounding add failures on solaris-sparcv9 and -server
500 java/math/BigDecimal/AddTests.java solaris-sparcv9

If this test is failing on a particular platform, that is indicative of
a HotSpot bug on that platform.

502 # Problems on windows with samevm, missing inputstream close()?
503 # Also times out on solaris-sparcv9 -server
504 java/math/BigInteger/BigIntegerTest.java generic-all

Some uses of input/output are not guarded by the try-finally {close
resource} structure. I've filed 6908541 "Bad resource management in
java/math/BigInteger/BigIntegerTest.java" for this.

505
506 # Should be samevm? But seems problematic with samevm on windows
507 java/math/BigInteger/ModPow65537.java

From a quick look, the test uses some security infrastructure, but
there is nothing that would seem problematic for samevm operation.

hg: jdk7/tl/jdk: 6903754: (bf) Improve floating-point buffer comparison

2009-12-08T12:59:11Z

Changeset: a0f6963b1618
Author: martin
Date: 2009-12-08 12:41 -0800
URL: http://hg.openjdk.java.net/jdk7/tl/jdk/rev/a0f6963b1618

6903754: (bf) Improve floating-point buffer comparison
Summary: Describe the exact behavior of {Double,Float}Buffer.{equals,compareTo}; fix non-anti-symmetric behavior of compareTo
Reviewed-by: alanb
Contributed-by: jessewilson@...

! make/java/nio/genBuffer.sh
! src/share/classes/java/nio/X-Buffer.java.template
! test/java/nio/Buffer/Basic-X.java.template
! test/java/nio/Buffer/BasicByte.java
! test/java/nio/Buffer/BasicChar.java
! test/java/nio/Buffer/BasicDouble.java
! test/java/nio/Buffer/BasicFloat.java
! test/java/nio/Buffer/BasicInt.java
! test/java/nio/Buffer/BasicLong.java
! test/java/nio/Buffer/BasicShort.java
! test/java/nio/Buffer/genBasic.sh

Re: Need reviewers, jdk7 testing changes

2009-12-08T11:19:48Z

Martin Buchholz wrote:

> On Mon, Dec 7, 2009 at 13:43, Tim Bell <Tim.Bell@...> wrote:
>>> java/util/Collection/MOAT.java
>> It is a shame we have to exclude Martin's MOAT.java (Mother Of All Tests)
>> The comment shows the test failing on Solaris 10 x86, but you exclude
>> it on generic-all
>
> You cannot exclude the Mother Of All Tests.
> This is non-negotiable.
>
> Martin

This is a problem list, and although I'm using it to exclude tests
from the jdk/test/Makefile jdk_* targets, there are multiple reasons
why a test is listed. If I'm wrong, I'll fix it.

I don't want to pick on your Mother test, and I will try and see why
it failed and de-list it if at all possible. Alan had similar issues
and so far we can't reproduce some of the nio test failures either.
I have to admit that it could have been "system fuzz" that caused
the failures in some cases. :^(

My initial attitude had been pretty strict, if a test failed when all
the other tests in it's batch passed, I had little mercy on it.
But from now on, I will see if I can reproduce the failure.
If I can reproduce the failure, and it appears to be a testcase issue,
I will be filing a bug.

I'll also be going back over the tests in the ProblemList and either
filing testcase bugs or jdk bugs, perhaps for batches of tests, not one bug
per testcase, my fellow developers might do me great bodily harm
if I filed hundreds of testcase bugs. :^(

-kto

Re: Need approval, JDI updates to openjdk6

2009-12-08T10:04:02Z

Joseph D. Darcy wrote:

> Kelly O'Hair wrote:
>>
>>
>> Joe Darcy wrote:
>>> Kelly O'Hair wrote:
>>>>
>>>> OpenJDK6 changes for JDI. I'm ready to push.
>>>>
>>>> I would like approvals from Joe, Jim, Alan, Dan, and Tim.
>>>
>>> Approved with comments.
>>>
>>> test/com/sun/jdi/StringConvertTest.sh
>>> 26 # @test @(#)StringConvertTest.sh 1.6 03/04/09
>>>
>>> An expanded sccs keywork crept back in!
>>>
>>> Otherwise, it looks fine.
>>>
>>> Do you think a diff of the files compared to JDK 7 would be informative?
>>
>> I actually did that on quite a few of the directories.
>> Well, diff -r -w that is. Ignored whitespace differences.
>>
>
> I thought you might say what the diffs were, "only non-whitespace diffs
> in ..." :-)
>
> -Joe
>

I went back and did another detailed diff between the latest in jdk7/tl and
what I have. I did miss a few items, namely the removal of -source 1.5
on the JDI tests, Copyright year updates, and I missed a jhat source file
change for the 2GB limit bug 6614052.

Now when I compare these directories:
src/*/classes/com/sun/tools/hat
src/*/classes/com/sun/tools/jdi
src/*/back
src/*/transport
src/*/demo/jvmti
test/com/sun/jdi
test/demo/jvmti
test/sun/tools/jhat

There are only 2 differences between the jdk7 and jdk6 files for JDI:

1. Solaris 8 stdint.h file missing change:

diff -w jdk7-tl/jdk/src/solaris/back/util_md.h src/solaris/back/util_md.h
30d29
< #include <stdint.h> /* To get uintptr_t */
31a31,40
> /* To get uintptr_t */
> #ifdef LINUX
> #include <stdint.h>
> #else
> /* The file stdint.h is not on Solaris 8 machines. */
> #include <sys/int_types.h>
> #include <sys/int_limits.h>
> #include <sys/int_const.h>
> #endif
>

2. Explicit difference for testing StackMap attributes:

diff -w jdk7-tl/jdk/test/demo/jvmti/hprof/StackMapTableTest.java test/demo/jvmti/hprof/StackMapTableTest.java
30c30
< * @compile -source 7 -g:lines HelloWorld.java
---
> * @compile -source 1.6 -g:lines HelloWorld.java

I updated the webrev at:
http://cr.openjdk.java.net/~ohair/openjdk6/jdk6-serviceability-update/webrev/

-kto

Re: Need approval, JDI updates to openjdk6

2009-12-08T09:05:35Z

Kelly O'Hair wrote:

>
>
> Joe Darcy wrote:
>> Kelly O'Hair wrote:
>>>
>>> OpenJDK6 changes for JDI. I'm ready to push.
>>>
>>> I would like approvals from Joe, Jim, Alan, Dan, and Tim.
>>
>> Approved with comments.
>>
>> test/com/sun/jdi/StringConvertTest.sh
>> 26 # @test @(#)StringConvertTest.sh 1.6 03/04/09
>>
>> An expanded sccs keywork crept back in!
>>
>> Otherwise, it looks fine.
>>
>> Do you think a diff of the files compared to JDK 7 would be informative?
>
> I actually did that on quite a few of the directories.
> Well, diff -r -w that is. Ignored whitespace differences.
>

I thought you might say what the diffs were, "only non-whitespace diffs
in ..." :-)

-Joe

Re: Need approval, JDI updates to openjdk6

2009-12-08T08:56:30Z

Joe Darcy wrote:

> Kelly O'Hair wrote:
>>
>> OpenJDK6 changes for JDI. I'm ready to push.
>>
>> I would like approvals from Joe, Jim, Alan, Dan, and Tim.
>
> Approved with comments.
>
> test/com/sun/jdi/StringConvertTest.sh
> 26 # @test @(#)StringConvertTest.sh 1.6 03/04/09
>
> An expanded sccs keywork crept back in!
>
> Otherwise, it looks fine.
>
> Do you think a diff of the files compared to JDK 7 would be informative?

I actually did that on quite a few of the directories.
Well, diff -r -w that is. Ignored whitespace differences.

-kto

>
> -Joe
>

Re: Need reviewers, jdk7 testing changes

2009-12-08T04:50:11Z

Kelly O'Hair wrote:
>
> :
>
> Dan had pointed out my typo in the jdk/test/Makefile ($->$$)
> but when I started running and re-running the tests I ran into
> more unstable tests. This webrev includes fixes to 3 tests
> and unfortuntately more additions to the ProblemList file.
It would be good to create a bug (or ensure there is an existing bug)
for any tests that you add to the list.
>
> :
> java/io/File/SetLastModified.java
I tried to duplicate this without success. Is the test directory on a
SAMBA drive when it fails?

> java/nio/channels/DatagramChannel/SRTest.java
> java/nio/channels/DatagramChannel/Sender.java
> java/nio/channels/Selector/SelectWrite.java
> java/nio/channels/DatagramChannel/EmptyBuffer.java
The older NIO test uses a fixed port which is problematic when tests run
in parallel. One of these days/weeks I will fix all these issues (they
manifest in various forms including hangs during initialization, or
address already in use errors).

> java/nio/channels/DatagramChannel/MulticastSendReceiveTests.java
This is an important test and I'd like to track down the reason it
failed for you rather than excluding it.

> java/util/Collection/MOAT.java
Over Martin's dead body :-)

> java/nio/channels/AsynchronousSocketChannel/Basic.java
Excluding this test will eliminate almost all test coverage of
AsynchronousSocketChannel so I don't think we should exclude it. I see
that the issue is a connection to a non-existent host is failing
immediately whereas the test expects it to timeout. I'll fix the test to
handle this case in b79.

> java/nio/channels/AsynchronousChannelGroup/Unbounded.java
As Windows 2000 Pro was EOL'ed eons ago we didn't put any effort into
the asynchronous I/O implementation to support that edition (it needs
Windows XP or newer). Can you just exclude this with windows-5.0 rather
than generic-all so that it continues to run on other platforms?

> java/nio/channels/FileChannel/Transfer.java
This test does take a long time on Windows. I'd be interested to see the
log as the right solution might be to increase the timeout or else just
exclude it on windows-5.0 (not generic-all as the test provides
important coverage).

-Alan.

hg: jdk7/tl/jdk: 6905046: More Dual-pivot quicksort improvements

2009-12-08T04:43:31Z

Changeset: 146c8921267f
Author: jjb
Date: 2009-12-08 12:40 +0000
URL: http://hg.openjdk.java.net/jdk7/tl/jdk/rev/146c8921267f

6905046: More Dual-pivot quicksort improvements
Summary: More improvements from the DPQ team
Reviewed-by: alanb

! src/share/classes/java/util/DualPivotQuicksort.java

Re: Need reviewers, jdk7 testing changes

2009-12-07T23:30:43Z

On Mon, Dec 7, 2009 at 13:43, Tim Bell <Tim.Bell@...> wrote:
>> java/util/Collection/MOAT.java
>
> It is a shame we have to exclude Martin's MOAT.java (Mother Of All Tests)
> The comment shows the test failing on Solaris 10 x86, but you exclude
> it on generic-all

You cannot exclude the Mother Of All Tests.
This is non-negotiable.

Martin

Re: Need approval, JDI updates to openjdk6

2009-12-07T18:39:53Z

Kelly O'Hair wrote:
>
> OpenJDK6 changes for JDI. I'm ready to push.
>
> I would like approvals from Joe, Jim, Alan, Dan, and Tim.

Approved with comments.

test/com/sun/jdi/StringConvertTest.sh
26 # @test @(#)StringConvertTest.sh 1.6 03/04/09

An expanded sccs keywork crept back in!

Otherwise, it looks fine.

Do you think a diff of the files compared to JDK 7 would be informative?

-Joe