Where are we with Surefire 2.4?

[dfabulich]

As you should all know by now, I've been on a hard sprint most of this
week to get a reasonable-looking Surefire 2.4 out the door.  I think we're
basically where I want it to be.

Just three main areas remain.

1) useSystemClassLoader issues
2) Snapshot dependencies
3) Measuring code coverage
and finally a remark 4) Dan takes a breather

1) useSystemClassLoader issues

Notably a lot of the very trickiest Surefire bugs we've gotten have all
been "Not Reproducible" or "Won't Fix", because all you have to do is set
useSystemClassLoader=true and things "just work."  I'd love to make that
the default setting for Surefire, but there's some stuff in the way that
worries me about doing that.

SUREFIRE-334 (targeted for 2.3.1) points out that we're depending on a
good chunk of plexus stuff in Surefire, and that this would infect the AUT
if we just set useSystemClassLoader=true.  332 and 347 are probably dupes
of this issue, but these things are subtle so I haven't closed them yet.

Jason suggested that we might use the maven-shade-plugin to shade our
plexus dependencies (repackaging them in different package names and
updating dependencies); it's a good idea, but maven-shade-plugin fails to
work properly on the plexus archiver (it doesn't translate interfaces
correctly, apparently?).

It would be great to resolve this issue and make useSystemClassLoader the
default BEFORE 2.4 ships (it *is* targeted for 2.3.1).  I think Brett has
claimed these.  (The other 2.3.1 bug 335 is a weird one, suggesting that
we might somehow useSystemClassLoader without forking; that sounds like
nonsense to me and I'd be curious to hear someone [Brett] comment on it.)

2) Snapshot dependencies (SUREFIRE-394)

2.4 depends on plexus-utils 1.4.8-SNAPSHOT; it should be released before
we release Surefire 2.4.

Up until last week or so we used to depend on plexus-archiver
alpha-10-SNAPSHOT, but that created ICCEs (see SUREFIRE-389) and so I
downgraded to alpha-7. If we DO need alpha-10-SNAPSHOT, then we'll need to
figure out the dependency issue (perhaps by fixing SUREFIRE-334 and
applying maven-shade-plugin?).  jdcasey apparently knew why we needed
alpha-10, but I still don't; my integration tests are all passing without
it.

3) Measuring code coverage

I wrote all these tests!  How good are they?  Measuring the coverage of
out-of-proc integration tests seems to be pretty tricky, though not
impossible.  Any ideas?

4) Dan takes a breather

I've been focused on getting this far all week, but I'll be basically
useless for most of the next week, maybe two.  I'll try to be available to
answer e-mail and chat, but this next week I won't be doing a lot of
coding.

I'd appreciate it if somebody non-me were to take up the baton and help
get this release out sometime this week (we're so close)! :-)

OK, that's it.  Let me know if you have questions.  I'm off to bed!

-Dan

[dfabulich]

I forgot, there's one more issue that should really get into Surefire 2.4:
SUREFIRE-303 http://jira.codehaus.org/browse/SUREFIRE-303 "Ignored/Skipped
tests are not reported"

Currently there's no standard on the ground for how JUnit 4 @Ignores
should be handled.  TestNG has a reasonable example (they put <skipped>
elements into the JUnit XML); we should, I suppose, follow their lead,
though I'm afraid the Ant/JUnit teams will decide to call it <ignored>
instead...

Still, I think we'd better clarify this before we try to support something
non-standard in Surefire 2.4.

-Dan

Dan Fabulich wrote:

>
> As you should all know by now, I've been on a hard sprint most of this week
> to get a reasonable-looking Surefire 2.4 out the door.  I think we're
> basically where I want it to be.
>
> Just three main areas remain.
>
> 1) useSystemClassLoader issues
> 2) Snapshot dependencies
> 3) Measuring code coverage
> and finally a remark 4) Dan takes a breather
>
> 1) useSystemClassLoader issues
>
> Notably a lot of the very trickiest Surefire bugs we've gotten have all been
> "Not Reproducible" or "Won't Fix", because all you have to do is set
> useSystemClassLoader=true and things "just work."  I'd love to make that the
> default setting for Surefire, but there's some stuff in the way that worries
> me about doing that.
>
> SUREFIRE-334 (targeted for 2.3.1) points out that we're depending on a good
> chunk of plexus stuff in Surefire, and that this would infect the AUT if we
> just set useSystemClassLoader=true.  332 and 347 are probably dupes of this
> issue, but these things are subtle so I haven't closed them yet.
>
> Jason suggested that we might use the maven-shade-plugin to shade our plexus
> dependencies (repackaging them in different package names and updating
> dependencies); it's a good idea, but maven-shade-plugin fails to work
> properly on the plexus archiver (it doesn't translate interfaces correctly,
> apparently?).
>
> It would be great to resolve this issue and make useSystemClassLoader the
> default BEFORE 2.4 ships (it *is* targeted for 2.3.1).  I think Brett has
> claimed these.  (The other 2.3.1 bug 335 is a weird one, suggesting that we
> might somehow useSystemClassLoader without forking; that sounds like nonsense
> to me and I'd be curious to hear someone [Brett] comment on it.)
>
> 2) Snapshot dependencies (SUREFIRE-394)
>
> 2.4 depends on plexus-utils 1.4.8-SNAPSHOT; it should be released before we
> release Surefire 2.4.
>
> Up until last week or so we used to depend on plexus-archiver
> alpha-10-SNAPSHOT, but that created ICCEs (see SUREFIRE-389) and so I
> downgraded to alpha-7. If we DO need alpha-10-SNAPSHOT, then we'll need to
> figure out the dependency issue (perhaps by fixing SUREFIRE-334 and applying
> maven-shade-plugin?).  jdcasey apparently knew why we needed alpha-10, but I
> still don't; my integration tests are all passing without it.
>
> 3) Measuring code coverage
>
> I wrote all these tests!  How good are they?  Measuring the coverage of
> out-of-proc integration tests seems to be pretty tricky, though not
> impossible.  Any ideas?
>
> 4) Dan takes a breather
>
> I've been focused on getting this far all week, but I'll be basically useless
> for most of the next week, maybe two.  I'll try to be available to answer
> e-mail and chat, but this next week I won't be doing a lot of coding.
>
> I'd appreciate it if somebody non-me were to take up the baton and help get
> this release out sometime this week (we're so close)! :-)
>
> OK, that's it.  Let me know if you have questions.  I'm off to bed!
>
> -Dan
>

[brettporter]

Hey Dan,

I concur on (1) and should have cycles this weekend on the flight  
back to wrap up 2.3.1 and hopefully take care of this for 2.4. You're  
right, the use of the two options together is non-sensical - which I  
think the issue was to just make it not attempt to let you use them  
together? As for p-archiver, I'm inclined to try and remove it since  
it's just assembling a very basic JAR.

For (2), as above.

I'll reply separately on (3) when I get a chance.

+1 for (4) - your efforts are very very much appreciated. Thanks!

Cheers,
Brett

On 26/11/2007, at 7:23 PM, Dan Fabulich wrote:

>
> As you should all know by now, I've been on a hard sprint most of  
> this week to get a reasonable-looking Surefire 2.4 out the door.  I  
> think we're basically where I want it to be.
>
> Just three main areas remain.
>
> 1) useSystemClassLoader issues
> 2) Snapshot dependencies
> 3) Measuring code coverage
> and finally a remark 4) Dan takes a breather
>
> 1) useSystemClassLoader issues
>
> Notably a lot of the very trickiest Surefire bugs we've gotten have  
> all been "Not Reproducible" or "Won't Fix", because all you have to  
> do is set useSystemClassLoader=true and things "just work."  I'd  
> love to make that the default setting for Surefire, but there's  
> some stuff in the way that worries me about doing that.
>
> SUREFIRE-334 (targeted for 2.3.1) points out that we're depending  
> on a good chunk of plexus stuff in Surefire, and that this would  
> infect the AUT if we just set useSystemClassLoader=true.  332 and  
> 347 are probably dupes of this issue, but these things are subtle  
> so I haven't closed them yet.
>
> Jason suggested that we might use the maven-shade-plugin to shade  
> our plexus dependencies (repackaging them in different package  
> names and updating dependencies); it's a good idea, but maven-shade-
> plugin fails to work properly on the plexus archiver (it doesn't  
> translate interfaces correctly, apparently?).
>
> It would be great to resolve this issue and make  
> useSystemClassLoader the default BEFORE 2.4 ships (it *is* targeted  
> for 2.3.1).  I think Brett has claimed these.  (The other 2.3.1 bug  
> 335 is a weird one, suggesting that we might somehow  
> useSystemClassLoader without forking; that sounds like nonsense to  
> me and I'd be curious to hear someone [Brett] comment on it.)
>
> 2) Snapshot dependencies (SUREFIRE-394)
>
> 2.4 depends on plexus-utils 1.4.8-SNAPSHOT; it should be released  
> before we release Surefire 2.4.
>
> Up until last week or so we used to depend on plexus-archiver  
> alpha-10-SNAPSHOT, but that created ICCEs (see SUREFIRE-389) and so  
> I downgraded to alpha-7. If we DO need alpha-10-SNAPSHOT, then  
> we'll need to figure out the dependency issue (perhaps by fixing  
> SUREFIRE-334 and applying maven-shade-plugin?).  jdcasey apparently  
> knew why we needed alpha-10, but I still don't; my integration  
> tests are all passing without it.
>
> 3) Measuring code coverage
>
> I wrote all these tests!  How good are they?  Measuring the  
> coverage of out-of-proc integration tests seems to be pretty  
> tricky, though not impossible.  Any ideas?
>
> 4) Dan takes a breather
>
> I've been focused on getting this far all week, but I'll be  
> basically useless for most of the next week, maybe two.  I'll try  
> to be available to answer e-mail and chat, but this next week I  
> won't be doing a lot of coding.
>
> I'd appreciate it if somebody non-me were to take up the baton and  
> help get this release out sometime this week (we're so close)! :-)
>
> OK, that's it.  Let me know if you have questions.  I'm off to bed!
>
> -Dan

--
Brett Porter - brett@...
Blog: http://www.devzuz.org/blogs/bporter/

[dfabulich]

Brett Porter wrote:

> As for p-archiver, I'm inclined to try and remove it since it's just
> assembling a very basic JAR.

Beware... manifest classpaths are surprisingly tricky to construct.  They
have a lot of weird corner cases you have to take care of.  (They must be
wrapped, wrapped lines must start with a space, but they're
space-delimited...)

But, yeah, I can't really object to removing it if that's all we need it
for.  We ARE using plexus-utils in a much more sophisticated way, (though
I bet it would be more amenable to the shade plugin...?) and might not
want to drop it.

Plus I just asked olamy to release a new version of plexus-utils for
Surefire, so I'd feel kinda bad if we turned around and dropped it.  ;-)

-Dan