RE: Continuing tests after a step failure

[Ian Homer]

Hi,

Just reading back catalog of topics on the mailing list having  
questioned to myself whether tests could continue a step failure and I  
found this thread.

Personally I think there is some value in having a soft step failure,  
i.e. a step failure that does not cause subsequent steps in a webtest  
to fail.  For example I have a test case that involves multiple forms  
and I want to verify the title of each form as I go through the  
flow.   I ran this test and it failed on the second form because the  
title on the second page was not correct.  That's fine - because  
that's a failed test, however this failure does not in reality prevent  
the functional flow from continuing, the second form can still be  
filled in and run OK.  Now I could out the tests into separate  
webtests, and I could write the steps as macrodefs (or Groovy closures  
as I'm writing tests in Groovy), but this seems an unnecessary  
abstraction when I have a single script which I'd like to cover  
various functional aspects.

For example my groovy test script might be written as ...

       webtest("test name") {
         group login
         verifyTitle 'Title 1'
         group inputForm1.curry(name:'Test',age:
65,sex:'Female',smoker:'No')
         verifyTitle 'Title 2'
        group inputForm2.curry(name:'Test',age:65,sex:'Male',smoker:'No')
         verifyText 'Title 3'
       }

(where login, inputForm1 and inputForm2 are closures defining how I  
log in and fill in the two forms)

Having a construct where I could identify one of the steps as a soft  
failure, e.g.

   soft { verifyTitle 'Title 2' }

would be extremely useful, keep my test as a single script and allow  
me to identify steps which do not break the flow on failure.

What's the general consensus on this and would this approach be  
something that would be considered?

Ian

--
Ian Homer
mobile ... made simple
http://bemoko.com | twitter: ianhomer











_______________________________________________
WebTest mailing list
WebTest@...
http://lists.canoo.com/mailman/listinfo/webtest

[John Spann]

Hi Ian,

This is actually very easy to implement, although we write xml tests with ant macros, which I don't know how to re-use in groovy tests.  You should be able to utilize the same logic inside your groovy tests, though.

First, we wrote a macro which allows for a soft failure:

<macrodef  name="softFail"  description="Wraps Steps such that the failures are ignored">
    <element name="steps" implicit="true"/>
    <sequential>
        <retry description="softFail Wrapper" maxcount="2" counterName="softFailCounter">
            <ifStep>
                <condition>
                    <verifyProperty description="Verify this is the first run" name="softFailCounter" value="0" propertyType="dynamic" />
                </condition>
                <then>
                    <steps/>
                </then>
                <else>
                    <setProperty description="Set softFailure to true so the test will fail" name="softFailure" value="true" propertyType="dynamic" />
                </else>
            </ifStep>
        </retry>    </sequential> </macrodef>

We then wrote a second macro which verifies no failure occurred:

<macrodef name="checkSoftFailure" description="Fail the test if a soft failure occurred">
    <sequential>
        <not>
            <verifyProperty description="Check if #{softFailure} is set to true" name="softFailure" value="true" propertyType="dynamic" />
        </not>
    </sequential>
</macrodef>

Our tests then look like this:

<webtest name="Soft Fail Example">
    <steps>
        <!-- Do stuff -->
        <softFail>
                <!--Step or steps that can fail -->
        </softFail>
        <!-- Do more stuff -->
        <checkSoftFailure/>
    </steps>
</webtest>

Hope that helps,

John Spann | Associate Software Engineer

Citrix Online Division
Citrix Systems, Inc.
6500 Hollister Avenue
Goleta, CA 93117 USA
www.citrix.com

Phone: [phone]
Cell: 805.729.0008
Email: John.Spann@...


________________________________
From: Ian Homer <ian.homer@...>
Reply-To: <webtest@...>, Ian Homer <ian.homer@...>
Date: Thu, 11 Jun 2009 01:17:48 -0700
To: <webtest@...>
Subject: [Webtest] RE: Continuing tests after a step failure

Hi,

Just reading back catalog of topics on the mailing list having
questioned to myself whether tests could continue a step failure and I
found this thread.

Personally I think there is some value in having a soft step failure,
i.e. a step failure that does not cause subsequent steps in a webtest
to fail.  For example I have a test case that involves multiple forms
and I want to verify the title of each form as I go through the
flow.   I ran this test and it failed on the second form because the
title on the second page was not correct.  That's fine - because
that's a failed test, however this failure does not in reality prevent
the functional flow from continuing, the second form can still be
filled in and run OK.  Now I could out the tests into separate
webtests, and I could write the steps as macrodefs (or Groovy closures
as I'm writing tests in Groovy), but this seems an unnecessary
abstraction when I have a single script which I'd like to cover
various functional aspects.

For example my groovy test script might be written as ...

       webtest("test name") {
         group login
         verifyTitle 'Title 1'
         group inputForm1.curry(name:'Test',age:
65,sex:'Female',smoker:'No')
         verifyTitle 'Title 2'
        group inputForm2.curry(name:'Test',age:65,sex:'Male',smoker:'No')
         verifyText 'Title 3'
       }

(where login, inputForm1 and inputForm2 are closures defining how I
log in and fill in the two forms)

Having a construct where I could identify one of the steps as a soft
failure, e.g.

   soft { verifyTitle 'Title 2' }

would be extremely useful, keep my test as a single script and allow
me to identify steps which do not break the flow on failure.

What's the general consensus on this and would this approach be
something that would be considered?

Ian

--
Ian Homer
mobile ... made simple
http://bemoko.com | twitter: ianhomer











_______________________________________________
WebTest mailing list
WebTest@...
http://lists.canoo.com/mailman/listinfo/webtest

_______________________________________________
WebTest mailing list
WebTest@...
http://lists.canoo.com/mailman/listinfo/webtest

[Ian Homer]

Nice one ... I've taken your example into the groovy world with ...

       def fullTestName="My Test"
       webtest() {
         group login
         group soft.curry("$fullTestName : Title 1", {verifyTitle  
'Title 1'})
         group inputForm1.curry(name:'Test',age:
65,sex:'Female',smoker:'No')
         group soft.curry("$fullTestName : Title 2", {verifyTitle  
'Title 2'})
         group inputForm2.curry(name:'Test',age:
65,sex:'Male',smoker:'No')
         group soft.curry("$fullTestName : Title 3", {verifyTitle  
'Title 3'})
       }
       verfifyNoSoftFailures()

All soft failures are stored whilst running the main webtest and then  
reported as failures in separate webtests.  I don't see this approach  
as particularly elegant.  For example the webtest reports are not so  
clear (because of the grouping). But it does the job for now.  It  
would be interesting to see if there is interest in an improvement to  
this technique.

The "soft" closure is defined as

    def soft={ description, steps ->
      groovy { log.info("Executing soft steps : $description ") }
      retry (maxcount:2,counterName:'i') {
        try {
          ifStep {
            condition { verifyProperty name:'i',value:0 }
            then { steps() }
          }
          ifStep {
            condition { not { verifyProperty name:'i',value:0 } }
            then { groovy { softFailures << description }  }
          }
        } catch (e) { log.error(e) }
      }
    }

(Side Note : I couldn't use the "else" step in groovy because it  
clashes with the groovy / java else syntax - anyone know if there is a  
way of addressing this conflict?)

and "verfifyNoSoftFailures" method is defined as

    def verfifyNoSoftFailures() {
      softFailures.each { description ->
        try {
          webtest(description) {
            storeProperty name:'fail',value:'false'
            verifyProperty description:"$description  
(failed)",name:'fail',value:'true'
          }
        } catch (e) { log.error(e) }
      }
      softFailures.clear()
    }

(Side Note: I couldn't find a simple step to trigger a failure hence  
I've done the verbose storeProperty/verifyProperty to trigger a  
failure - anyone know how to to do this in a cleaner way?)

and I have a class variable "softFailures" to store the soft failures

   def softFailures=[]

Cheers,

Ian

On 11 Jun 2009, at 19:27, John Spann wrote:

> Hi Ian,
>
> This is actually very easy to implement, although we write xml tests  
> with ant macros, which I don't know how to re-use in groovy tests.  
> You should be able to utilize the same logic inside your groovy  
> tests, though.
>
> First, we wrote a macro which allows for a soft failure:
>
> <macrodef  name="softFail"  description="Wraps Steps such that the  
> failures are ignored">
>    <element name="steps" implicit="true"/>
>    <sequential>
>        <retry description="softFail Wrapper" maxcount="2"  
> counterName="softFailCounter">
>            <ifStep>
>                <condition>
>                    <verifyProperty description="Verify this is the  
> first run" name="softFailCounter" value="0" propertyType="dynamic" />
>                </condition>
>                <then>
>                    <steps/>
>                </then>
>                <else>
>                    <setProperty description="Set softFailure to true  
> so the test will fail" name="softFailure" value="true"  
> propertyType="dynamic" />
>                </else>
>            </ifStep>
>        </retry>    </sequential> </macrodef>
>
> We then wrote a second macro which verifies no failure occurred:
>
> <macrodef name="checkSoftFailure" description="Fail the test if a  
> soft failure occurred">
>    <sequential>
>        <not>
>            <verifyProperty description="Check if #{softFailure} is  
> set to true" name="softFailure" value="true" propertyType="dynamic" />
>        </not>
>    </sequential>
> </macrodef>
>
> Our tests then look like this:
>
> <webtest name="Soft Fail Example">
>    <steps>
>        <!-- Do stuff -->
>        <softFail>
>                <!--Step or steps that can fail -->
>        </softFail>
>        <!-- Do more stuff -->
>        <checkSoftFailure/>
>    </steps>
> </webtest>
>
> Hope that helps,
>
> John Spann | Associate Software Engineer
>
> Citrix Online Division
> Citrix Systems, Inc.
> 6500 Hollister Avenue
> Goleta, CA 93117 USA
> www.citrix.com
>
> Phone: [phone]
> Cell: 805.729.0008
> Email: John.Spann@...
>
>
> ________________________________
> From: Ian Homer <ian.homer@...>
> Reply-To: <webtest@...>, Ian Homer <ian.homer@...>
> Date: Thu, 11 Jun 2009 01:17:48 -0700
> To: <webtest@...>
> Subject: [Webtest] RE: Continuing tests after a step failure
>
> Hi,
>
> Just reading back catalog of topics on the mailing list having
> questioned to myself whether tests could continue a step failure and I
> found this thread.
>
> Personally I think there is some value in having a soft step failure,
> i.e. a step failure that does not cause subsequent steps in a webtest
> to fail.  For example I have a test case that involves multiple forms
> and I want to verify the title of each form as I go through the
> flow.   I ran this test and it failed on the second form because the
> title on the second page was not correct.  That's fine - because
> that's a failed test, however this failure does not in reality prevent
> the functional flow from continuing, the second form can still be
> filled in and run OK.  Now I could out the tests into separate
> webtests, and I could write the steps as macrodefs (or Groovy closures
> as I'm writing tests in Groovy), but this seems an unnecessary
> abstraction when I have a single script which I'd like to cover
> various functional aspects.
>
> For example my groovy test script might be written as ...
>
>       webtest("test name") {
>         group login
>         verifyTitle 'Title 1'
>         group inputForm1.curry(name:'Test',age:
> 65,sex:'Female',smoker:'No')
>         verifyTitle 'Title 2'
>        group inputForm2.curry(name:'Test',age:
> 65,sex:'Male',smoker:'No')
>         verifyText 'Title 3'
>       }
>
> (where login, inputForm1 and inputForm2 are closures defining how I
> log in and fill in the two forms)
>
> Having a construct where I could identify one of the steps as a soft
> failure, e.g.
>
>   soft { verifyTitle 'Title 2' }
>
> would be extremely useful, keep my test as a single script and allow
> me to identify steps which do not break the flow on failure.
>
> What's the general consensus on this and would this approach be
> something that would be considered?
>
> Ian
>
> --
> Ian Homer
> mobile ... made simple
> http://bemoko.com | twitter: ianhomer
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> WebTest mailing list
> WebTest@...
> http://lists.canoo.com/mailman/listinfo/webtest
>

--
Ian Homer
mobile ... made simple
http://bemoko.com | twitter: ianhomer











_______________________________________________
WebTest mailing list
WebTest@...
http://lists.canoo.com/mailman/listinfo/webtest

[Marc Guillemot]

Hi,

@Ian: to simplify your tests, you can override the webtest method to
perform verfifyNoSoftFailures() after executing the closure you pass as
argument.

Some time ago (one or two years if I correctly remember) we already had
a similar discussion in this list. I think too that it could be
interesting to have the possibility to "collect" verification errors and
make the test fail at the end. The xml report format needs to be
enhanced for that to be able to store more than one failure per test.
As always: a patch is welcome ;-)

Cheers,
Marc.
--
Web: http://www.efficient-webtesting.com
Blog: http://mguillem.wordpress.com

Ian Homer wrote:

> Nice one ... I've taken your example into the groovy world with ...
>
>       def fullTestName="My Test"
>       webtest() {
>         group login
>         group soft.curry("$fullTestName : Title 1", {verifyTitle 'Title
> 1'})
>         group inputForm1.curry(name:'Test',age:65,sex:'Female',smoker:'No')
>         group soft.curry("$fullTestName : Title 2", {verifyTitle 'Title
> 2'})
>         group inputForm2.curry(name:'Test',age:65,sex:'Male',smoker:'No')
>         group soft.curry("$fullTestName : Title 3", {verifyTitle 'Title
> 3'})
>       }
>       verfifyNoSoftFailures()
>
> All soft failures are stored whilst running the main webtest and then
> reported as failures in separate webtests.  I don't see this approach as
> particularly elegant.  For example the webtest reports are not so clear
> (because of the grouping). But it does the job for now.  It would be
> interesting to see if there is interest in an improvement to this
> technique.
>
> The "soft" closure is defined as
>
>    def soft={ description, steps ->
>      groovy { log.info("Executing soft steps : $description ") }
>      retry (maxcount:2,counterName:'i') {
>        try {
>          ifStep {
>            condition { verifyProperty name:'i',value:0 }
>            then { steps() }
>          }
>          ifStep {
>            condition { not { verifyProperty name:'i',value:0 } }
>            then { groovy { softFailures << description }  }
>          }
>        } catch (e) { log.error(e) }
>      }
>    }
>
> (Side Note : I couldn't use the "else" step in groovy because it clashes
> with the groovy / java else syntax - anyone know if there is a way of
> addressing this conflict?)
>
> and "verfifyNoSoftFailures" method is defined as
>
>    def verfifyNoSoftFailures() {
>      softFailures.each { description ->
>        try {
>          webtest(description) {
>            storeProperty name:'fail',value:'false'
>            verifyProperty description:"$description
> (failed)",name:'fail',value:'true'
>          }
>        } catch (e) { log.error(e) }
>      }
>      softFailures.clear()
>    }
>
> (Side Note: I couldn't find a simple step to trigger a failure hence
> I've done the verbose storeProperty/verifyProperty to trigger a failure
> - anyone know how to to do this in a cleaner way?)
>
> and I have a class variable "softFailures" to store the soft failures
>
>   def softFailures=[]
>
> Cheers,
>
> Ian
>
> On 11 Jun 2009, at 19:27, John Spann wrote:
>
>> Hi Ian,
>>
>> This is actually very easy to implement, although we write xml tests
>> with ant macros, which I don't know how to re-use in groovy tests.
>> You should be able to utilize the same logic inside your groovy tests,
>> though.
>>
>> First, we wrote a macro which allows for a soft failure:
>>
>> <macrodef  name="softFail"  description="Wraps Steps such that the
>> failures are ignored">
>>    <element name="steps" implicit="true"/>
>>    <sequential>
>>        <retry description="softFail Wrapper" maxcount="2"
>> counterName="softFailCounter">
>>            <ifStep>
>>                <condition>
>>                    <verifyProperty description="Verify this is the
>> first run" name="softFailCounter" value="0" propertyType="dynamic" />
>>                </condition>
>>                <then>
>>                    <steps/>
>>                </then>
>>                <else>
>>                    <setProperty description="Set softFailure to true
>> so the test will fail" name="softFailure" value="true"
>> propertyType="dynamic" />
>>                </else>
>>            </ifStep>
>>        </retry>    </sequential> </macrodef>
>>
>> We then wrote a second macro which verifies no failure occurred:
>>
>> <macrodef name="checkSoftFailure" description="Fail the test if a soft
>> failure occurred">
>>    <sequential>
>>        <not>
>>            <verifyProperty description="Check if #{softFailure} is set
>> to true" name="softFailure" value="true" propertyType="dynamic" />
>>        </not>
>>    </sequential>
>> </macrodef>
>>
>> Our tests then look like this:
>>
>> <webtest name="Soft Fail Example">
>>    <steps>
>>        <!-- Do stuff -->
>>        <softFail>
>>                <!--Step or steps that can fail -->
>>        </softFail>
>>        <!-- Do more stuff -->
>>        <checkSoftFailure/>
>>    </steps>
>> </webtest>
>>
>> Hope that helps,
>>
>> John Spann | Associate Software Engineer
>>
>> Citrix Online Division
>> Citrix Systems, Inc.
>> 6500 Hollister Avenue
>> Goleta, CA 93117 USA
>> www.citrix.com
>>
>> Phone: [phone]
>> Cell: 805.729.0008
>> Email: John.Spann@...
>>
>>
>> ________________________________
>> From: Ian Homer <ian.homer@...>
>> Reply-To: <webtest@...>, Ian Homer <ian.homer@...>
>> Date: Thu, 11 Jun 2009 01:17:48 -0700
>> To: <webtest@...>
>> Subject: [Webtest] RE: Continuing tests after a step failure
>>
>> Hi,
>>
>> Just reading back catalog of topics on the mailing list having
>> questioned to myself whether tests could continue a step failure and I
>> found this thread.
>>
>> Personally I think there is some value in having a soft step failure,
>> i.e. a step failure that does not cause subsequent steps in a webtest
>> to fail.  For example I have a test case that involves multiple forms
>> and I want to verify the title of each form as I go through the
>> flow.   I ran this test and it failed on the second form because the
>> title on the second page was not correct.  That's fine - because
>> that's a failed test, however this failure does not in reality prevent
>> the functional flow from continuing, the second form can still be
>> filled in and run OK.  Now I could out the tests into separate
>> webtests, and I could write the steps as macrodefs (or Groovy closures
>> as I'm writing tests in Groovy), but this seems an unnecessary
>> abstraction when I have a single script which I'd like to cover
>> various functional aspects.
>>
>> For example my groovy test script might be written as ...
>>
>>       webtest("test name") {
>>         group login
>>         verifyTitle 'Title 1'
>>         group inputForm1.curry(name:'Test',age:
>> 65,sex:'Female',smoker:'No')
>>         verifyTitle 'Title 2'
>>        group inputForm2.curry(name:'Test',age:65,sex:'Male',smoker:'No')
>>         verifyText 'Title 3'
>>       }
>>
>> (where login, inputForm1 and inputForm2 are closures defining how I
>> log in and fill in the two forms)
>>
>> Having a construct where I could identify one of the steps as a soft
>> failure, e.g.
>>
>>   soft { verifyTitle 'Title 2' }
>>
>> would be extremely useful, keep my test as a single script and allow
>> me to identify steps which do not break the flow on failure.
>>
>> What's the general consensus on this and would this approach be
>> something that would be considered?
>>
>> Ian
>>
>> --
>> Ian Homer
>> mobile ... made simple
>> http://bemoko.com | twitter: ianhomer
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> WebTest mailing list
>> WebTest@...
>> http://lists.canoo.com/mailman/listinfo/webtest
>>
>

_______________________________________________
WebTest mailing list
WebTest@...
http://lists.canoo.com/mailman/listinfo/webtest

[Ian Homer]

Hi Marc,

On 16 Jun 2009, at 10:28, Marc Guillemot wrote:

> Hi,
>
> @Ian: to simplify your tests, you can override the webtest method to
> perform verfifyNoSoftFailures() after executing the closure you pass  
> as
> argument.

thx - good tip

>
> Some time ago (one or two years if I correctly remember) we already  
> had
> a similar discussion in this list. I think too that it could be
> interesting to have the possibility to "collect" verification errors  
> and
> make the test fail at the end. The xml report format needs to be
> enhanced for that to be able to store more than one failure per test.
> As always: a patch is welcome ;-)
>

I'll let my canoo skills settle a bit and then I might give it a go,  
it sounds an interesting challenge.

> Cheers,
> Marc.
> --
> Web: http://www.efficient-webtesting.com
> Blog: http://mguillem.wordpress.com
>
> Ian Homer wrote:
>> Nice one ... I've taken your example into the groovy world with ...
>>
>>      def fullTestName="My Test"
>>      webtest() {
>>        group login
>>        group soft.curry("$fullTestName : Title 1", {verifyTitle  
>> 'Title
>> 1'})
>>        group inputForm1.curry(name:'Test',age:
>> 65,sex:'Female',smoker:'No')
>>        group soft.curry("$fullTestName : Title 2", {verifyTitle  
>> 'Title
>> 2'})
>>        group inputForm2.curry(name:'Test',age:
>> 65,sex:'Male',smoker:'No')
>>        group soft.curry("$fullTestName : Title 3", {verifyTitle  
>> 'Title
>> 3'})
>>      }
>>      verfifyNoSoftFailures()
>>
>> All soft failures are stored whilst running the main webtest and then
>> reported as failures in separate webtests.  I don't see this  
>> approach as
>> particularly elegant.  For example the webtest reports are not so  
>> clear
>> (because of the grouping). But it does the job for now.  It would be
>> interesting to see if there is interest in an improvement to this
>> technique.
>>
>> The "soft" closure is defined as
>>
>>   def soft={ description, steps ->
>>     groovy { log.info("Executing soft steps : $description ") }
>>     retry (maxcount:2,counterName:'i') {
>>       try {
>>         ifStep {
>>           condition { verifyProperty name:'i',value:0 }
>>           then { steps() }
>>         }
>>         ifStep {
>>           condition { not { verifyProperty name:'i',value:0 } }
>>           then { groovy { softFailures << description }  }
>>         }
>>       } catch (e) { log.error(e) }
>>     }
>>   }
>>
>> (Side Note : I couldn't use the "else" step in groovy because it  
>> clashes
>> with the groovy / java else syntax - anyone know if there is a way of
>> addressing this conflict?)
>>
>> and "verfifyNoSoftFailures" method is defined as
>>
>>   def verfifyNoSoftFailures() {
>>     softFailures.each { description ->
>>       try {
>>         webtest(description) {
>>           storeProperty name:'fail',value:'false'
>>           verifyProperty description:"$description
>> (failed)",name:'fail',value:'true'
>>         }
>>       } catch (e) { log.error(e) }
>>     }
>>     softFailures.clear()
>>   }
>>
>> (Side Note: I couldn't find a simple step to trigger a failure hence
>> I've done the verbose storeProperty/verifyProperty to trigger a  
>> failure
>> - anyone know how to to do this in a cleaner way?)
>>
>> and I have a class variable "softFailures" to store the soft failures
>>
>>  def softFailures=[]
>>
>> Cheers,
>>
>> Ian
>>
>> On 11 Jun 2009, at 19:27, John Spann wrote:
>>
>>> Hi Ian,
>>>
>>> This is actually very easy to implement, although we write xml tests
>>> with ant macros, which I don't know how to re-use in groovy tests.
>>> You should be able to utilize the same logic inside your groovy  
>>> tests,
>>> though.
>>>
>>> First, we wrote a macro which allows for a soft failure:
>>>
>>> <macrodef  name="softFail"  description="Wraps Steps such that the
>>> failures are ignored">
>>>   <element name="steps" implicit="true"/>
>>>   <sequential>
>>>       <retry description="softFail Wrapper" maxcount="2"
>>> counterName="softFailCounter">
>>>           <ifStep>
>>>               <condition>
>>>                   <verifyProperty description="Verify this is the
>>> first run" name="softFailCounter" value="0"  
>>> propertyType="dynamic" />
>>>               </condition>
>>>               <then>
>>>                   <steps/>
>>>               </then>
>>>               <else>
>>>                   <setProperty description="Set softFailure to true
>>> so the test will fail" name="softFailure" value="true"
>>> propertyType="dynamic" />
>>>               </else>
>>>           </ifStep>
>>>       </retry>    </sequential> </macrodef>
>>>
>>> We then wrote a second macro which verifies no failure occurred:
>>>
>>> <macrodef name="checkSoftFailure" description="Fail the test if a  
>>> soft
>>> failure occurred">
>>>   <sequential>
>>>       <not>
>>>           <verifyProperty description="Check if #{softFailure} is  
>>> set
>>> to true" name="softFailure" value="true" propertyType="dynamic" />
>>>       </not>
>>>   </sequential>
>>> </macrodef>
>>>
>>> Our tests then look like this:
>>>
>>> <webtest name="Soft Fail Example">
>>>   <steps>
>>>       <!-- Do stuff -->
>>>       <softFail>
>>>               <!--Step or steps that can fail -->
>>>       </softFail>
>>>       <!-- Do more stuff -->
>>>       <checkSoftFailure/>
>>>   </steps>
>>> </webtest>
>>>
>>> Hope that helps,
>>>
>>> John Spann | Associate Software Engineer
>>>
>>> Citrix Online Division
>>> Citrix Systems, Inc.
>>> 6500 Hollister Avenue
>>> Goleta, CA 93117 USA
>>> www.citrix.com
>>>
>>> Phone: [phone]
>>> Cell: 805.729.0008
>>> Email: John.Spann@...
>>>
>>>
>>> ________________________________
>>> From: Ian Homer <ian.homer@...>
>>> Reply-To: <webtest@...>, Ian Homer <ian.homer@...>
>>> Date: Thu, 11 Jun 2009 01:17:48 -0700
>>> To: <webtest@...>
>>> Subject: [Webtest] RE: Continuing tests after a step failure
>>>
>>> Hi,
>>>
>>> Just reading back catalog of topics on the mailing list having
>>> questioned to myself whether tests could continue a step failure  
>>> and I
>>> found this thread.
>>>
>>> Personally I think there is some value in having a soft step  
>>> failure,
>>> i.e. a step failure that does not cause subsequent steps in a  
>>> webtest
>>> to fail.  For example I have a test case that involves multiple  
>>> forms
>>> and I want to verify the title of each form as I go through the
>>> flow.   I ran this test and it failed on the second form because the
>>> title on the second page was not correct.  That's fine - because
>>> that's a failed test, however this failure does not in reality  
>>> prevent
>>> the functional flow from continuing, the second form can still be
>>> filled in and run OK.  Now I could out the tests into separate
>>> webtests, and I could write the steps as macrodefs (or Groovy  
>>> closures
>>> as I'm writing tests in Groovy), but this seems an unnecessary
>>> abstraction when I have a single script which I'd like to cover
>>> various functional aspects.
>>>
>>> For example my groovy test script might be written as ...
>>>
>>>      webtest("test name") {
>>>        group login
>>>        verifyTitle 'Title 1'
>>>        group inputForm1.curry(name:'Test',age:
>>> 65,sex:'Female',smoker:'No')
>>>        verifyTitle 'Title 2'
>>>       group inputForm2.curry(name:'Test',age:
>>> 65,sex:'Male',smoker:'No')
>>>        verifyText 'Title 3'
>>>      }
>>>
>>> (where login, inputForm1 and inputForm2 are closures defining how I
>>> log in and fill in the two forms)
>>>
>>> Having a construct where I could identify one of the steps as a soft
>>> failure, e.g.
>>>
>>>  soft { verifyTitle 'Title 2' }
>>>
>>> would be extremely useful, keep my test as a single script and allow
>>> me to identify steps which do not break the flow on failure.
>>>
>>> What's the general consensus on this and would this approach be
>>> something that would be considered?
>>>
>>> Ian
>>>
>>> --
>>> Ian Homer
>>> mobile ... made simple
>>> http://bemoko.com | twitter: ianhomer
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> WebTest mailing list
>>> WebTest@...
>>> http://lists.canoo.com/mailman/listinfo/webtest
>>>
>>
>
> _______________________________________________
> WebTest mailing list
> WebTest@...
> http://lists.canoo.com/mailman/listinfo/webtest

--
Ian Homer
mobile ... made simple
http://bemoko.com | http://twitter.com/ianhomer












_______________________________________________
WebTest mailing list
WebTest@...
http://lists.canoo.com/mailman/listinfo/webtest