Performing checks and skipping on failure

[Matt Steele]

Hi all,

I'm just starting adding dependencies around my test groups and ran
into a scenario that I don't know how to model.

I'm executing my tests with TestNG 5.8 (jdk15) via the maven-surefire-
plugin 2.4.3.

I have a set of integration tests that are dependent on the
availability of external web services.  What I'd like to add is a test
to check for the successful response of the web service. If that check
fails, skip the integration tests.  So I can do this with a
combination of test groups and the dependsOnGroups parameter.

But what I'd also like to do is report the check as *skipped*, rather
than *failed*.

My reasoning is that we currently receive an email report from our CI
server whenever there is a build failure. But if an external service
that I have no control over fails, I would rather be blissfully
ignorant, and just execute my unit tests. Then, when the service
becomes available again, resume executing the integration tests.

Looking over a few of the mailing list threads, I couldn't find this
specific scenario, though this discussion looked fairly similar:
http://groups.google.com/group/testng-users/browse_thread/thread/9d85a1c49dcde2e7/86c01b18d828c3a6?lnk=gst&q=check+skip#86c01b18d828c3a6

Is there a way to accomplish this? I can add in my pom.xml snippits if
they're helpful.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng-users@...
To unsubscribe from this group, send email to testng-users+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

[Nithya Sakthirajan]

You can throw org.testng.SkipException in your code and tests will be
reported as skipped

Thanks
Nithya
On Wed, Oct 21, 2009 at 9:53 AM, Matt Steele <orphum@...> wrote:

>
> Hi all,
>
> I'm just starting adding dependencies around my test groups and ran
> into a scenario that I don't know how to model.
>
> I'm executing my tests with TestNG 5.8 (jdk15) via the maven-surefire-
> plugin 2.4.3.
>
> I have a set of integration tests that are dependent on the
> availability of external web services.  What I'd like to add is a test
> to check for the successful response of the web service. If that check
> fails, skip the integration tests.  So I can do this with a
> combination of test groups and the dependsOnGroups parameter.
>
> But what I'd also like to do is report the check as *skipped*, rather
> than *failed*.
>
> My reasoning is that we currently receive an email report from our CI
> server whenever there is a build failure. But if an external service
> that I have no control over fails, I would rather be blissfully
> ignorant, and just execute my unit tests. Then, when the service
> becomes available again, resume executing the integration tests.
>
> Looking over a few of the mailing list threads, I couldn't find this
> specific scenario, though this discussion looked fairly similar:
> http://groups.google.com/group/testng-users/browse_thread/thread/9d85a1c49dcde2e7/86c01b18d828c3a6?lnk=gst&q=check+skip#86c01b18d828c3a6
>
> Is there a way to accomplish this? I can add in my pom.xml snippits if
> they're helpful.
>
> >
>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng-users@...
To unsubscribe from this group, send email to testng-users+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

[Matt Steele]

That does the trick. Thanks!

On Wed, Oct 21, 2009 at 8:12 PM, Nithya Sakthirajan
<nithyasakthirajan@...> wrote:

>
> You can throw org.testng.SkipException in your code and tests will be
> reported as skipped
>
> Thanks
> Nithya
> On Wed, Oct 21, 2009 at 9:53 AM, Matt Steele <orphum@...> wrote:
>>
>> Hi all,
>>
>> I'm just starting adding dependencies around my test groups and ran
>> into a scenario that I don't know how to model.
>>
>> I'm executing my tests with TestNG 5.8 (jdk15) via the maven-surefire-
>> plugin 2.4.3.
>>
>> I have a set of integration tests that are dependent on the
>> availability of external web services.  What I'd like to add is a test
>> to check for the successful response of the web service. If that check
>> fails, skip the integration tests.  So I can do this with a
>> combination of test groups and the dependsOnGroups parameter.
>>
>> But what I'd also like to do is report the check as *skipped*, rather
>> than *failed*.
>>
>> My reasoning is that we currently receive an email report from our CI
>> server whenever there is a build failure. But if an external service
>> that I have no control over fails, I would rather be blissfully
>> ignorant, and just execute my unit tests. Then, when the service
>> becomes available again, resume executing the integration tests.
>>
>> Looking over a few of the mailing list threads, I couldn't find this
>> specific scenario, though this discussion looked fairly similar:
>> http://groups.google.com/group/testng-users/browse_thread/thread/9d85a1c49dcde2e7/86c01b18d828c3a6?lnk=gst&q=check+skip#86c01b18d828c3a6
>>
>> Is there a way to accomplish this? I can add in my pom.xml snippits if
>> they're helpful.
>>
>> >
>>
>
> >
>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng-users@...
To unsubscribe from this group, send email to testng-users+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en
-~----------~----~----~----~------~----~------~--~---

[gyrm]

I have a variation on this scenario which is currently does not seem
to be supported.

The method containing the SkipException logic is in a base class,
which all actual test classes subclass. This method calls into a
method which may be implemented / overridden by each test subclass
which actually determines whether the test is runnable, given the
information in the context object (which can change, especially across
threads). When the SkipException is thrown, however, because it is the
base class that is doing the throwing, all subsequent subclasses are
skipped!

Here's some Groovy code to illustrate (example only):


class Base {

    @BeforeClass(alwaysRun = true)
    void verifyRunnable(ITestContext context) {
        if (! isRunnableIn(context)) {
            throw new SkipException("Oops, couldn't run in $
{context}!")
        }

        // further operations
    }

    abstract bool isRunnableIn(ITestContext context)
}

class Derived1 extends Base {

    @Override
    bool isRunnableIn(ITestContext context) {
        // maybe returns true, or maybe returns false
    }

    @Test
    void testA() {
        // a test
    }
}

class Derived2 extends Base {

    @Override
    bool isRunnableIn(ITestContext context) {
        // maybe returns true, or maybe returns false
    }

    @Test
    void testB() {
        // a test
    }
}


Assuming Derived1 runs before Derived2, if its isRunnableIn() returns
false, then a SkipException is thrown by the base class, and Derived2
is skipped. This is not the desired behavior.

I'd like to know if people think this sort of scenario should be
supported, perhaps by introducing some kind of marker exception (or
additional property on SkipException) that informs TestNG of a
"targeted skip", if you will - to communicate that the configuration
method failure only the current class, and that other classes /
methods that depend on it should still be run. I'd also be interested
in hearing any obvious workarounds that I haven't thought of!  :)


Thanks,
Haw-Bin




On Oct 22, 9:33 am, Matt Steele <orp...@...> wrote:

> That does the trick. Thanks!
>
> On Wed, Oct 21, 2009 at 8:12 PM, Nithya Sakthirajan
>
>
>
> <nithyasakthira...@...> wrote:
>
> > You can throw org.testng.SkipException in your code and tests will be
> > reported as skipped
>
> > Thanks
> > Nithya
> > On Wed, Oct 21, 2009 at 9:53 AM, Matt Steele <orp...@...> wrote:
>
> >> Hi all,
>
> >> I'm just starting adding dependencies around my test groups and ran
> >> into a scenario that I don't know how to model.
>
> >> I'm executing my tests with TestNG 5.8 (jdk15) via the maven-surefire-
> >> plugin 2.4.3.
>
> >> I have a set of integration tests that are dependent on the
> >> availability of external web services.  What I'd like to add is a test
> >> to check for the successful response of the web service. If that check
> >> fails, skip the integration tests.  So I can do this with a
> >> combination of test groups and the dependsOnGroups parameter.
>
> >> But what I'd also like to do is report the check as *skipped*, rather
> >> than *failed*.
>
> >> My reasoning is that we currently receive an email report from our CI
> >> server whenever there is a build failure. But if an external service
> >> that I have no control over fails, I would rather be blissfully
> >> ignorant, and just execute my unit tests. Then, when the service
> >> becomes available again, resume executing the integration tests.
>
> >> Looking over a few of the mailing list threads, I couldn't find this
> >> specific scenario, though this discussion looked fairly similar:
> >>http://groups.google.com/group/testng-users/browse_thread/thread/9d85...
>
> >> Is there a way to accomplish this? I can add in my pom.xml snippits if
> >> they're helpful.- Hide quoted text -
>
> - Show quoted text -

--

You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng-users@....
To unsubscribe from this group, send email to testng-users+unsubscribe@....
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

[Cédric Beust ♔]

Hi Haw-Bin,

The problem is that you are throwing this exception in a configuration method, so this behavior is intentional.

You should try to move the "isRunnable" test into a test method, for example with a dependency.

Did you try this?

--

Cédric

On Sat, Dec 5, 2009 at 3:56 PM, gyrm <hbchai@...> wrote:

I have a variation on this scenario which is currently does not seem
to be supported.

The method containing the SkipException logic is in a base class,
which all actual test classes subclass. This method calls into a
method which may be implemented / overridden by each test subclass
which actually determines whether the test is runnable, given the
information in the context object (which can change, especially across
threads). When the SkipException is thrown, however, because it is the
base class that is doing the throwing, all subsequent subclasses are
skipped!

Here's some Groovy code to illustrate (example only):


class Base {

   @BeforeClass(alwaysRun = true)
   void verifyRunnable(ITestContext context) {
       if (! isRunnableIn(context)) {
           throw new SkipException("Oops, couldn't run in $
{context}!")
       }

       // further operations
   }

   abstract bool isRunnableIn(ITestContext context)
}

class Derived1 extends Base {

   @Override
   bool isRunnableIn(ITestContext context) {
       // maybe returns true, or maybe returns false
   }

   @Test
   void testA() {
       // a test
   }
}

class Derived2 extends Base {

   @Override
   bool isRunnableIn(ITestContext context) {
       // maybe returns true, or maybe returns false
   }

   @Test
   void testB() {
       // a test
   }
}


Assuming Derived1 runs before Derived2, if its isRunnableIn() returns
false, then a SkipException is thrown by the base class, and Derived2
is skipped. This is not the desired behavior.

I'd like to know if people think this sort of scenario should be
supported, perhaps by introducing some kind of marker exception (or
additional property on SkipException) that informs TestNG of a
"targeted skip", if you will - to communicate that the configuration
method failure only the current class, and that other classes /
methods that depend on it should still be run. I'd also be interested
in hearing any obvious workarounds that I haven't thought of!  :)


Thanks,
Haw-Bin

On Oct 22, 9:33 am, Matt Steele <orp...@...> wrote:
> That does the trick. Thanks!
>
> On Wed, Oct 21, 2009 at 8:12 PM, Nithya Sakthirajan
>
>
>

> <nithyasakthira...@...> wrote:
>
> > You can throw org.testng.SkipException in your code and tests will be
> > reported as skipped
>
> > Thanks
> > Nithya

> > On Wed, Oct 21, 2009 at 9:53 AM, Matt Steele <orp...@...> wrote:
>
> >> Hi all,
>
> >> I'm just starting adding dependencies around my test groups and ran
> >> into a scenario that I don't know how to model.
>
> >> I'm executing my tests with TestNG 5.8 (jdk15) via the maven-surefire-
> >> plugin 2.4.3.
>
> >> I have a set of integration tests that are dependent on the
> >> availability of external web services.  What I'd like to add is a test
> >> to check for the successful response of the web service. If that check
> >> fails, skip the integration tests.  So I can do this with a
> >> combination of test groups and the dependsOnGroups parameter.
>
> >> But what I'd also like to do is report the check as *skipped*, rather
> >> than *failed*.
>
> >> My reasoning is that we currently receive an email report from our CI
> >> server whenever there is a build failure. But if an external service
> >> that I have no control over fails, I would rather be blissfully
> >> ignorant, and just execute my unit tests. Then, when the service
> >> becomes available again, resume executing the integration tests.
>
> >> Looking over a few of the mailing list threads, I couldn't find this
> >> specific scenario, though this discussion looked fairly similar:

> >>http://groups.google.com/group/testng-users/browse_thread/thread/9d85...

>
> >> Is there a way to accomplish this? I can add in my pom.xml snippits if

> >> they're helpful.- Hide quoted text -
>
> - Show quoted text -

--

You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng-users@....
To unsubscribe from this group, send email to testng-users%2Bunsubscribe@....

For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--

You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng-users@....
To unsubscribe from this group, send email to testng-users+unsubscribe@....
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

[gyrm]

Hi Cedric,

I wanted to avoid adding additional @Test methods and duplicating the
throwing of SkipException if possible.

I have, however, come up with a working solution, using
IInvokedMethodListener!  :)  Throwing the SkipException in
beforeInvocation() results in the desired behavior - the test is
marked as skipped, while others depending on the @BeforeClass (which
no longer throws) are still run. beforeInvocation() simply checks the
condition result, which was actually calculated in the @BeforeClass.
Thus test classes need only override isRunnableIn(), if desired.

This is the second time IInvokedMethodListener has saved me!


Haw-Bin


On Dec 5, 7:34 pm, Cédric Beust ♔ <cbe...@...> wrote:

> Hi Haw-Bin,
>
> The problem is that you are throwing this exception in a configuration
> method, so this behavior is intentional.
>
> You should try to move the "isRunnable" test into a test method, for example
> with a dependency.
>
> Did you try this?
>
> --
> ***Cédric
> *
>
> On Sat, Dec 5, 2009 at 3:56 PM, gyrm <hbc...@...> wrote:
> > I have a variation on this scenario which is currently does not seem
> > to be supported.
>
> > The method containing the SkipException logic is in a base class,
> > which all actual test classes subclass. This method calls into a
> > method which may be implemented / overridden by each test subclass
> > which actually determines whether the test is runnable, given the
> > information in the context object (which can change, especially across
> > threads). When the SkipException is thrown, however, because it is the
> > base class that is doing the throwing, all subsequent subclasses are
> > skipped!
>
> > Here's some Groovy code to illustrate (example only):
>
> > class Base {
>
> >    @BeforeClass(alwaysRun = true)
> >    void verifyRunnable(ITestContext context) {
> >        if (! isRunnableIn(context)) {
> >            throw new SkipException("Oops, couldn't run in $
> > {context}!")
> >        }
>
> >        // further operations
> >    }
>
> >    abstract bool isRunnableIn(ITestContext context)
> > }
>
> > class Derived1 extends Base {
>
> >    @Override
> >    bool isRunnableIn(ITestContext context) {
> >        // maybe returns true, or maybe returns false
> >    }
>
> >    @Test
> >    void testA() {
> >        // a test
> >    }
> > }
>
> > class Derived2 extends Base {
>
> >    @Override
> >    bool isRunnableIn(ITestContext context) {
> >        // maybe returns true, or maybe returns false
> >    }
>
> >    @Test
> >    void testB() {
> >        // a test
> >    }
> > }
>
> > Assuming Derived1 runs before Derived2, if its isRunnableIn() returns
> > false, then a SkipException is thrown by the base class, and Derived2
> > is skipped. This is not the desired behavior.
>
> > I'd like to know if people think this sort of scenario should be
> > supported, perhaps by introducing some kind of marker exception (or
> > additional property on SkipException) that informs TestNG of a
> > "targeted skip", if you will - to communicate that the configuration
> > method failure only the current class, and that other classes /
> > methods that depend on it should still be run. I'd also be interested
> > in hearing any obvious workarounds that I haven't thought of!  :)
>
> > Thanks,
> > Haw-Bin
>
> > On Oct 22, 9:33 am, Matt Steele <orp...@...> wrote:
> > > That does the trick. Thanks!
>
> > > On Wed, Oct 21, 2009 at 8:12 PM, Nithya Sakthirajan
>
> > > <nithyasakthira...@...> wrote:
>
> > > > You can throw org.testng.SkipException in your code and tests will be
> > > > reported as skipped
>
> > > > Thanks
> > > > Nithya
> > > > On Wed, Oct 21, 2009 at 9:53 AM, Matt Steele <orp...@...> wrote:
>
> > > >> Hi all,
>
> > > >> I'm just starting adding dependencies around my test groups and ran
> > > >> into a scenario that I don't know how to model.
>
> > > >> I'm executing my tests with TestNG 5.8 (jdk15) via the maven-surefire-
> > > >> plugin 2.4.3.
>
> > > >> I have a set of integration tests that are dependent on the
> > > >> availability of external web services.  What I'd like to add is a test
> > > >> to check for the successful response of the web service. If that check
> > > >> fails, skip the integration tests.  So I can do this with a
> > > >> combination of test groups and the dependsOnGroups parameter.
>
> > > >> But what I'd also like to do is report the check as *skipped*, rather
> > > >> than *failed*.
>
> > > >> My reasoning is that we currently receive an email report from our CI
> > > >> server whenever there is a build failure. But if an external service
> > > >> that I have no control over fails, I would rather be blissfully
> > > >> ignorant, and just execute my unit tests. Then, when the service
> > > >> becomes available again, resume executing the integration tests.
>
> > > >> Looking over a few of the mailing list threads, I couldn't find this
> > > >> specific scenario, though this discussion looked fairly similar:
> > > >>http://groups.google.com/group/testng-users/browse_thread/thread/9d85.
> > ..
>
> > > >> Is there a way to accomplish this? I can add in my pom.xml snippits if
> > > >> they're helpful.- Hide quoted text -
>
> > > - Show quoted text -
>
> > --
>
> > You received this message because you are subscribed to the Google Groups
> > "testng-users" group.
> > To post to this group, send email to testng-users@....
> > To unsubscribe from this group, send email to
> > testng-users+unsubscribe@...<testng-users%2Bunsubscribe@...>
> > .
> > For more options, visit this group at
> >http://groups.google.com/group/testng-users?hl=en.

--

You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng-users@....
To unsubscribe from this group, send email to testng-users+unsubscribe@....
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.