Testing in multi-project builds

[Russel Winder-4]

I am setting Gant up as a four project build with each project using the
same source code -- just different Groovy dependencies.  It seems though
that by default a multi-project test fails immediately after the first
failing project.  Is this intended?  Is there an easy way of getting the
test to always run all tests?

Thanks.

--
Russel.
=============================================================================
Dr Russel Winder      Partner
                                            xmpp: russel@...
Concertant LLP        t: [phone], [phone]
41 Buckmaster Road,   f: [phone]   voip: sip:russel.winder@...
London SW11 1EN, UK   m: [phone]   skype: russel_winder

[levi_h]

Hi Russel,

which test framework are you using? With TestNG you van use the  
failOnError property; JUnit probably has something similar, but I  
haven't used it for years.

   Levi

Op 4 okt 2009 om 12:33 heeft Russel Winder  
<russel.winder@...> het volgende geschreven:\

> I am setting Gant up as a four project build with each project using  
> the
> same source code -- just different Groovy dependencies.  It seems  
> though
> that by default a multi-project test fails immediately after the first
> failing project.  Is this intended?  Is there an easy way of getting  
> the
> test to always run all tests?
>
> Thanks.
>
> --
> Russel.
> ===
> ===
> ===
> ====================================================================
> Dr Russel Winder      Partner
>                                            xmpp: russel@...
> Concertant LLP        t: [phone], [phone]
> 41 Buckmaster Road,   f: [phone]   voip: sip:russel.winder@...
> London SW11 1EN, UK   m: [phone]   skype: russel_winder

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Russel Winder-4]

Levi,

On Sun, 2009-10-04 at 13:25 +0200, Levi Hoogenberg wrote:
> Hi Russel,
>
> which test framework are you using? With TestNG you van use the  
> failOnError property; JUnit probably has something similar, but I  
> haven't used it for years.
>
>    Levi

In this case I am using GroovyTestCases so JUnit under the covers.
Whilst I prefer TestNG when using a framework directly, Groovy favours
JUnit.

Though I am not sure failOnError is an issue here.  The tests for a
given project are failing and Gradle is deciding to terminate the
multi-project build on the first test fail.  I suspect what I need is a
way of telling Gradle to ignore unit test fails.

Or maybe I am just misunderstanding the relationship Gradle sets up with
the unit test runner -- which is entirely possible :-)
 
--
Russel.
=============================================================================
Dr Russel Winder      Partner
                                            xmpp: russel@...
Concertant LLP        t: [phone], [phone]
41 Buckmaster Road,   f: [phone]   voip: sip:russel.winder@...
London SW11 1EN, UK   m: [phone]   skype: russel_winder

[levi_h]

So, if I understand you correctly, if one of the subprojects' tests fail, you would like the build to continue, but still like it to fail in the end?

On Sun, Oct 4, 2009 at 2:29 PM, Russel Winder <russel.winder@...> wrote:

Levi,

On Sun, 2009-10-04 at 13:25 +0200, Levi Hoogenberg wrote:
> Hi Russel,
>
> which test framework are you using? With TestNG you van use the
> failOnError property; JUnit probably has something similar, but I
> haven't used it for years.
>
>    Levi

In this case I am using GroovyTestCases so JUnit under the covers.
Whilst I prefer TestNG when using a framework directly, Groovy favours
JUnit.

Though I am not sure failOnError is an issue here.  The tests for a
given project are failing and Gradle is deciding to terminate the
multi-project build on the first test fail.  I suspect what I need is a
way of telling Gradle to ignore unit test fails.

Or maybe I am just misunderstanding the relationship Gradle sets up with
the unit test runner -- which is entirely possible :-)

--
Russel.
=============================================================================
Dr Russel Winder      Partner
                                           xmpp: russel@...
Concertant LLP        t: [phone], [phone]
41 Buckmaster Road,   f: [phone]   voip: sip%3Arussel.winder@...
London SW11 1EN, UK   m: [phone]   skype: russel_winder

[Russel Winder-4]

On Sun, 2009-10-04 at 18:53 +0200, Levi Hoogenberg wrote:
> So, if I understand you correctly, if one of the subprojects' tests
> fail, you would like the build to continue, but still like it to fail
> in the end?
>
Yes.

The requirement here is to be able to run all the tests for all the
projects in a CI system.  So, as you sat, a project's test fail
shouldn't stop the other projects tests running but the final return
code needs to be failure.

> On Sun, Oct 4, 2009 at 2:29 PM, Russel Winder
> <russel.winder@...> wrote:
>         Levi,
>        
>         On Sun, 2009-10-04 at 13:25 +0200, Levi Hoogenberg wrote:
>         > Hi Russel,
>         >
>         > which test framework are you using? With TestNG you van use
>         the
>         > failOnError property; JUnit probably has something similar,
>         but I
>         > haven't used it for years.
>         >
>         >    Levi
>        
>        
>         In this case I am using GroovyTestCases so JUnit under the
>         covers.
>         Whilst I prefer TestNG when using a framework directly, Groovy
>         favours
>         JUnit.
>        
>         Though I am not sure failOnError is an issue here.  The tests
>         for a
>         given project are failing and Gradle is deciding to terminate
>         the
>         multi-project build on the first test fail.  I suspect what I
>         need is a
>         way of telling Gradle to ignore unit test fails.
>        
>         Or maybe I am just misunderstanding the relationship Gradle
>         sets up with
>         the unit test runner -- which is entirely possible :-)
>        
>        
>         --
>         Russel.
>         =============================================================================
>         Dr Russel Winder      Partner
>                                                    xmpp:
>         russel@...
>         Concertant LLP        t: [phone], [phone]
>         41 Buckmaster Road,   f: [phone]   voip:
>         sip:russel.winder@...
>         London SW11 1EN, UK   m: [phone]   skype:
>         russel_winder
>        
>

--

Russel.
=============================================================================
Dr Russel Winder      Partner
                                            xmpp: russel@...
Concertant LLP        t: [phone], [phone]
41 Buckmaster Road,   f: [phone]   voip: sip:russel.winder@...
London SW11 1EN, UK   m: [phone]   skype: russel_winder

[levi_h]

In that case, the JUnit equivalent of failOnError could still help to continue the build. The other part could be trickier.

My idea was to check for test failures for one of your subprojects at the end of your build (or the end of your CI task). For TestNG, you could check for the existence of testng-failed.xml. Maybe with JUnit there's something similar? Then, when you discover test failures, you could fail the build. I'm not sure what's the best way to do that, though - throwing an exception would certainly work :)

On Sun, Oct 4, 2009 at 7:31 PM, Russel Winder <russel.winder@...> wrote:

On Sun, 2009-10-04 at 18:53 +0200, Levi Hoogenberg wrote:
> So, if I understand you correctly, if one of the subprojects' tests
> fail, you would like the build to continue, but still like it to fail
> in the end?
>

Yes.

The requirement here is to be able to run all the tests for all the
projects in a CI system.  So, as you sat, a project's test fail
shouldn't stop the other projects tests running but the final return
code needs to be failure.

> On Sun, Oct 4, 2009 at 2:29 PM, Russel Winder
> <russel.winder@...> wrote:
>         Levi,
>
>         On Sun, 2009-10-04 at 13:25 +0200, Levi Hoogenberg wrote:
>         > Hi Russel,
>         >
>         > which test framework are you using? With TestNG you van use
>         the
>         > failOnError property; JUnit probably has something similar,
>         but I
>         > haven't used it for years.
>         >
>         >    Levi
>
>
>         In this case I am using GroovyTestCases so JUnit under the
>         covers.
>         Whilst I prefer TestNG when using a framework directly, Groovy
>         favours
>         JUnit.
>
>         Though I am not sure failOnError is an issue here.  The tests
>         for a
>         given project are failing and Gradle is deciding to terminate
>         the
>         multi-project build on the first test fail.  I suspect what I
>         need is a
>         way of telling Gradle to ignore unit test fails.
>
>         Or maybe I am just misunderstanding the relationship Gradle
>         sets up with
>         the unit test runner -- which is entirely possible :-)
>
>
>         --
>         Russel.
>         =============================================================================
>         Dr Russel Winder      Partner
>                                                    xmpp:
>         russel@...
>         Concertant LLP        t: [phone], [phone]
>         41 Buckmaster Road,   f: [phone]   voip:
>         sip%3Arussel.winder@...
>         London SW11 1EN, UK   m: [phone]   skype:
>         russel_winder
>
>

--

Russel.
=============================================================================
Dr Russel Winder      Partner
                                           xmpp: russel@...
Concertant LLP        t: [phone], [phone]
41 Buckmaster Road,   f: [phone]   voip: sip%3Arussel.winder@...
London SW11 1EN, UK   m: [phone]   skype: russel_winder

[hdockter]

On Oct 4, 2009, at 7:31 PM, Russel Winder wrote:

> On Sun, 2009-10-04 at 18:53 +0200, Levi Hoogenberg wrote:
>> So, if I understand you correctly, if one of the subprojects' tests
>> fail, you would like the build to continue, but still like it to fail
>> in the end?
>>
> Yes.
>
> The requirement here is to be able to run all the tests for all the
> projects in a CI system.  So, as you sat, a project's test fail
> shouldn't stop the other projects tests running but the final return
> code needs to be failure.

The new native test runners we developing for 0.9 should provide a  
nice solution for this.

Meanwhile you could do the following. Set stopAtFailuresOrErrors to  
true and add the following hook:

gradle.taskGraph.afterTask { task, exception ->
    if (task instanceof Test && exception != null) {
       // do something (for example set a property)
    }
}

Have a special ci task that let the build fail if any tests in the  
subprojects have failed.

For 0.9 we will have an api method of the test task that tells you  
whether tests have failed or not.

- Hans

--
Hans Dockter
Gradle Project Manager
http://www.gradle.org



---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Russel Winder-4]

Levi,

On Mon, 2009-10-05 at 07:16 +0200, Levi Hoogenberg wrote:

> In that case, the JUnit equivalent of failOnError could still help to
> continue the build. The other part could be trickier.
>
> My idea was to check for test failures for one of your subprojects at
> the end of your build (or the end of your CI task). For TestNG, you
> could check for the existence of testng-failed.xml. Maybe with JUnit
> there's something similar? Then, when you discover test failures, you
> could fail the build. I'm not sure what's the best way to do that,
> though - throwing an exception would certainly work :)
>

Thanks for the ideas.  I'll take Hans' advice (see next email) on first

though and see if that works.


--
Russel.
=============================================================================
Dr Russel Winder      Partner
                                            xmpp: russel@...
Concertant LLP        t: [phone], [phone]
41 Buckmaster Road,   f: [phone]   voip: sip:russel.winder@...
London SW11 1EN, UK   m: [phone]   skype: russel_winder

[Russel Winder-4]

Hans,

On Mon, 2009-10-05 at 13:01 +0200, Hans Dockter wrote:
[ . . . ]
> The new native test runners we developing for 0.9 should provide a  
> nice solution for this.

OK, I need to be an early adopter I think !

The application is the Gant build system which is currently a 4 project
build but may end up as a 5 project one -- each project is simply a
different build of the same source (not actually a different project).
What I want to do is to ensure that all project tests are run in the
Bamboo and Buildbot CI builds even if individual ones fail.

> Meanwhile you could do the following. Set stopAtFailuresOrErrors to  
> true and add the following hook:
>
> gradle.taskGraph.afterTask { task, exception ->
>     if (task instanceof Test && exception != null) {
>        // do something (for example set a property)
>     }
> }
>
> Have a special ci task that let the build fail if any tests in the  
> subprojects have failed.
>
> For 0.9 we will have an api method of the test task that tells you  
> whether tests have failed or not.

Thanks, I'll give this a go tomorrow.

--
Russel.
=============================================================================
Dr Russel Winder      Partner
                                            xmpp: russel@...
Concertant LLP        t: [phone], [phone]
41 Buckmaster Road,   f: [phone]   voip: sip:russel.winder@...
London SW11 1EN, UK   m: [phone]   skype: russel_winder