Surefire 2.4 is coming

[dfabulich]

Surefire trunk currently depends on a couple of SNAPSHOT versions that we
expect to be released in the next couple of days, but other than that, as
far as I know, there's nothing else standing in the way of a Surefire 2.4
release.  It includes more than 50 bug fixes.

With that said, the more testing we get for Surefire 2.4 the better,
because (as I understand it) people are going to be auto-upgraded to
Surefire 2.4 when it gets released; when Surefire 2.3 came out, it wasn't
uncommon to hear people complaining that their tests used to work earlier
in the week, but post-2.3 no longer work.

The easiest way you can help us test Surefire is to pull down surefire
trunk and do "mvn clean install"; that will run all of the Surefire
integration tests.  If any of those fail, please file a bug with a
stacktrace (and the log.txt from your surefire-integration-tests/target
directory, if applicable).

https://svn.apache.org/repos/asf/maven/surefire/trunk

The second easiest way you can help test Surefire 2.4 is to try the latest
SNAPSHOT out in various projects you support.  If you find that Surefire
2.4-SNAPSHOT breaks anything, file bugs and we'll try to address them.
(Reduced test cases in the form of minimal Maven projects that reproduce
the problem would be greatly appreciated.)

-Dan

[brettporter]

Thanks Dan!

I'm just catching up again. I am planning to scan through all the  
commits and see if anything catches my eye. I also saw an issue  
reported on the users list that I'll drop into JIRA about the new code  
working with 2.0.6 - which versions of Maven have you been testing  
with? (I've tended to test with 2.0.4 and 2.0.6, but now 2.0.7 and  
2.0.8 made sometimes funky changes that are worth running the  
integration tests with).

WDYT about continuing with the 2.3.1 release immediately, and giving  
2.4 a short time more to simmer? If for any reason 2.4 does break  
certain builds, there is still a better-than-2.3 fallback. You can  
also use the release ann. to re-encourage testing of 2.4.

And again, thanks - the quantity of work you did here was just  
awesome :)

- Brett

On 11/12/2007, at 10:25 AM, Dan Fabulich wrote:

>
> Surefire trunk currently depends on a couple of SNAPSHOT versions  
> that we expect to be released in the next couple of days, but other  
> than that, as far as I know, there's nothing else standing in the  
> way of a Surefire 2.4 release.  It includes more than 50 bug fixes.
>
> With that said, the more testing we get for Surefire 2.4 the better,  
> because (as I understand it) people are going to be auto-upgraded to  
> Surefire 2.4 when it gets released; when Surefire 2.3 came out, it  
> wasn't uncommon to hear people complaining that their tests used to  
> work earlier in the week, but post-2.3 no longer work.
>
> The easiest way you can help us test Surefire is to pull down  
> surefire trunk and do "mvn clean install"; that will run all of the  
> Surefire integration tests.  If any of those fail, please file a bug  
> with a stacktrace (and the log.txt from your surefire-integration-
> tests/target directory, if applicable).
>
> https://svn.apache.org/repos/asf/maven/surefire/trunk
>
> The second easiest way you can help test Surefire 2.4 is to try the  
> latest SNAPSHOT out in various projects you support.  If you find  
> that Surefire 2.4-SNAPSHOT breaks anything, file bugs and we'll try  
> to address them. (Reduced test cases in the form of minimal Maven  
> projects that reproduce the problem would be greatly appreciated.)
>
> -Dan

[dfabulich]

Brett Porter wrote:

> I'm just catching up again. I am planning to scan through all the commits and
> see if anything catches my eye. I also saw an issue reported on the users
> list that I'll drop into JIRA about the new code working with 2.0.6 - which
> versions of Maven have you been testing with? (I've tended to test with 2.0.4
> and 2.0.6, but now 2.0.7 and 2.0.8 made sometimes funky changes that are
> worth running the integration tests with).

I've been pretty much exclusively testing with 2.0.7.  I just ran the 2.4
integration tests with 2.0.6 and 2.0.8 with no problems.

maven-surefire-plugin lists 2.0.6 as a prerequisite, as of revision
598160.  "mkleint: plexus-utils version 1.4.x means the plugins requires
2.0.6 at least".  Not entirely certain why that is, but there you have it.

> WDYT about continuing with the 2.3.1 release immediately, and giving 2.4
> a short time more to simmer? If for any reason 2.4 does break certain
> builds, there is still a better-than-2.3 fallback. You can also use the
> release ann. to re-encourage testing of 2.4.

+0 on letting 2.4 simmer; -0 on releasing 2.3.1 now.  I think I wouldn't
want to let 2.4 simmer for more than two weeks, at the maximum.

What concerns me about 2.3.1 is that it doesn't have integration tests
like 2.4, so it's hard to know for sure whether it really *is* better than
2.3 or whether new bugs have creeped in.  (Surefire has a lot of corner
cases and gotchas.)

On the other hand, I feel pretty good about 2.4-SNAPSHOT because it has
been getting a lot of incidental testing through users who want/need
TestNG support.  TestNG recommends using 2.4-SNAPSHOT on their website;
pretty much everyone using TestNG and Maven has been using 2.4-SNAPSHOT
all along.

If Surefire didn't automatically grab the latest release, I'd cheerfully
agree to release both and let users sort out what works best for them, but
as it stands I'd prefer to destabilize Surefire infrequently and put all
of our testing eggs in one basket.

With that said, if there's a need for 2.3.1 [Mauro suggested he'd like it
last week :-)] then I certainly won't stand in its way.  I'd strongly
suggest that whoever wanted to try this should run the 2.4 integration
tests against 2.3 and then again against 2.3.1 to identify regressions.
Seems like a lot of trouble for not much benefit, IMO.

-Dan

[brettporter]

On 12/12/2007, at 6:52 AM, Dan Fabulich wrote:

> Brett Porter wrote:
>
>> I'm just catching up again. I am planning to scan through all the  
>> commits and see if anything catches my eye. I also saw an issue  
>> reported on the users list that I'll drop into JIRA about the new  
>> code working with 2.0.6 - which versions of Maven have you been  
>> testing with? (I've tended to test with 2.0.4 and 2.0.6, but now  
>> 2.0.7 and 2.0.8 made sometimes funky changes that are worth running  
>> the integration tests with).
>
> I've been pretty much exclusively testing with 2.0.7.  I just ran  
> the 2.4 integration tests with 2.0.6 and 2.0.8 with no problems.
>
> maven-surefire-plugin lists 2.0.6 as a prerequisite, as of revision  
> 598160.  "mkleint: plexus-utils version 1.4.x means the plugins  
> requires 2.0.6 at least".  Not entirely certain why that is, but  
> there you have it.

2.0.6 was the first version to shade the plexus-utils dependency away  
from the plugins.

>
>
>> WDYT about continuing with the 2.3.1 release immediately, and  
>> giving 2.4 a short time more to simmer? If for any reason 2.4 does  
>> break certain builds, there is still a better-than-2.3 fallback.  
>> You can also use the release ann. to re-encourage testing of 2.4.
>
> +0 on letting 2.4 simmer; -0 on releasing 2.3.1 now.  I think I  
> wouldn't want to let 2.4 simmer for more than two weeks, at the  
> maximum.
>
> What concerns me about 2.3.1 is that it doesn't have integration  
> tests like 2.4, so it's hard to know for sure whether it really *is*  
> better than 2.3 or whether new bugs have creeped in.  (Surefire has  
> a lot of corner cases and gotchas.)

That was kind of the intent of 2.3.1 - limited fixes for known  
regressions and small issues in 2.3... at least with the work I did on  
it I felt pretty confident. I see your point though, and I'm happy  
either way, just thought I'd throw the suggestion out there.

Cheers,
Brett