testng, surefire, and forkMode=always

[dfabulich]

Benjamin Bentmann has filed bug SUREFIRE-446
http://jira.codehaus.org/browse/SUREFIRE-446

Maven Surefire provides a facility for various "fork modes":

* forkMode=once: The default mode, spawns a separate TestNG process (like
the TestNG Ant task)

* forkMode=never: run TestNG in the same process as Maven, in an isolated
classloader

* forkMode=always: launch TestNG N times for N tests; if one test
segfaults or calls System.exit or something, the other tests can run to
completion safely.

I didn't test forkMode=always with TestNG in Surefire 2.4, and sure enough
it doesn't work at all.  (It runs the tests but reports 0 tests
passed/failed.)

It does work basically OK for JUnit, where classes are normally isolated
from each other, but I'm not sure it even makes sense for TestNG tests,
where one test may depend on tests in other classes (multi-class dependent
methods).

I can kind of hack it so it works OK if you don't have multi-class
dependent methods, but if you DO have multi-class dependent methods, it
fails with a TestNGException: "MyTest.testFoo() is depending on
nonexistent method otherClassSetUp".

What should I do here?  Put in the hack and let the user figure out the
TestNGException if they try to use multi-class dependent methods?  Fail
with a clean error if the user tries to use forkMode=always with TestNG?
Something else?

Should forkMode=always be a feature of TestNG somehow?

-Dan

[Cédric Beust ♔]

Creating a TestNG instance for every @Test method?  Seems completely useless
to me.

Especially when you can control the threading behavior with the parallel
attribute.

--
Cédric


On Wed, Feb 6, 2008 at 2:59 PM, Dan Fabulich <dan@...> wrote:

>
>
> Benjamin Bentmann has filed bug SUREFIRE-446
> http://jira.codehaus.org/browse/SUREFIRE-446
>
> Maven Surefire provides a facility for various "fork modes":
>
> * forkMode=once: The default mode, spawns a separate TestNG process (like
> the TestNG Ant task)
>
> * forkMode=never: run TestNG in the same process as Maven, in an isolated
> classloader
>
> * forkMode=always: launch TestNG N times for N tests; if one test
> segfaults or calls System.exit or something, the other tests can run to
> completion safely.
>
> I didn't test forkMode=always with TestNG in Surefire 2.4, and sure enough
> it doesn't work at all.  (It runs the tests but reports 0 tests
> passed/failed.)
>
> It does work basically OK for JUnit, where classes are normally isolated
> from each other, but I'm not sure it even makes sense for TestNG tests,
> where one test may depend on tests in other classes (multi-class dependent
> methods).
>
> I can kind of hack it so it works OK if you don't have multi-class
> dependent methods, but if you DO have multi-class dependent methods, it
> fails with a TestNGException: "MyTest.testFoo() is depending on
> nonexistent method otherClassSetUp".
>
> What should I do here?  Put in the hack and let the user figure out the
> TestNGException if they try to use multi-class dependent methods?  Fail
> with a clean error if the user tries to use forkMode=always with TestNG?
> Something else?
>
> Should forkMode=always be a feature of TestNG somehow?
>
> -Dan
>
> --~--~---------~--~----~------------~-------~--~----~
> You received this message because you are subscribed to the Google Groups
> "testng-users" group.
> To post to this group, send email to testng-users@...
> To unsubscribe from this group, send email to
> testng-users-unsubscribe@...
> For more options, visit this group at
> http://groups.google.com/group/testng-users?hl=en
> -~----------~----~----~----~------~----~------~--~---
>
>

[brettporter]

My inclination would be towards just an error that says its not  
supported under TestNG.

I don't know why people even use it with JUnit, except when they have  
memory leaks in their tests, or tests that affect the state of the JVM  
that they don't tear down :)

- Brett

On 07/02/2008, at 9:59 AM, Dan Fabulich wrote:

>
> Benjamin Bentmann has filed bug SUREFIRE-446 http://jira.codehaus.org/browse/SUREFIRE-446
>
> Maven Surefire provides a facility for various "fork modes":
>
> * forkMode=once: The default mode, spawns a separate TestNG process  
> (like the TestNG Ant task)
>
> * forkMode=never: run TestNG in the same process as Maven, in an  
> isolated classloader
>
> * forkMode=always: launch TestNG N times for N tests; if one test  
> segfaults or calls System.exit or something, the other tests can run  
> to completion safely.
>
> I didn't test forkMode=always with TestNG in Surefire 2.4, and sure  
> enough it doesn't work at all.  (It runs the tests but reports 0  
> tests passed/failed.)
>
> It does work basically OK for JUnit, where classes are normally  
> isolated from each other, but I'm not sure it even makes sense for  
> TestNG tests, where one test may depend on tests in other classes  
> (multi-class dependent methods).
>
> I can kind of hack it so it works OK if you don't have multi-class  
> dependent methods, but if you DO have multi-class dependent methods,  
> it fails with a TestNGException: "MyTest.testFoo() is depending on  
> nonexistent method otherClassSetUp".
>
> What should I do here?  Put in the hack and let the user figure out  
> the TestNGException if they try to use multi-class dependent  
> methods?  Fail with a clean error if the user tries to use  
> forkMode=always with TestNG? Something else?
>
> Should forkMode=always be a feature of TestNG somehow?
>
> -Dan

[Benjamin Bentmann]

Cédric Beust ♔ wrote:

Creating a TestNG instance for every @Test method?  Seems completely useless
to me.

The intent is not to fork once per test method but once per test class. Just like I can say in the TestNG Eclipse plugin "run class XY" instead of selecting a group.


Benjamin Bentmann

[Benjamin Bentmann]

dfabulich wrote:

What should I do here?  Put in the hack and let the user figure out the
TestNGException if they try to use multi-class dependent methods?

Please, give us the exception from TestNG if the setup is invalid, give us the mere test results if the setup is valid because no inter-class dependencies exist. I mean somebody who can setup a config file for a build tool is smart enough to figure out what an exception means or do you disagree? Just in case, a few lines of doc about the forkMode=always setting should successfully shield Surefire from users filling in unjustified issues.

To me, it seems this discussion is primarily concerned about making Surefire bullet-proof against bad user config. As for me, I have this slightly arrogant attitude to feel superior to a machine and hence always feel kind of angry if a program wants to tell what I should better not do because it might fail.


Benjamin Bentmann

[Benjamin Bentmann]

brettporter wrote:

I don't know why people even use it with JUnit

I am not really after forking all my tests, they already take long enough... As Dan already guessed, I was more after a way to hack me through to single-class test-suites. If forkMode=always would have worked, I would have argued, "well, I want this behavior but without the JVM forking, please" ;-)


Benjamin Bentmann