Re: RE: Continuing tests after a step failure

> Nice one ... I've taken your example into the groovy world with ...
>
>       def fullTestName="My Test"
>       webtest() {
>         group login
>         group soft.curry("$fullTestName : Title 1", {verifyTitle 'Title
> 1'})
>         group inputForm1.curry(name:'Test',age:65,sex:'Female',smoker:'No')
>         group soft.curry("$fullTestName : Title 2", {verifyTitle 'Title
> 2'})
>         group inputForm2.curry(name:'Test',age:65,sex:'Male',smoker:'No')
>         group soft.curry("$fullTestName : Title 3", {verifyTitle 'Title
> 3'})
>       }
>       verfifyNoSoftFailures()
>
> All soft failures are stored whilst running the main webtest and then
> reported as failures in separate webtests.  I don't see this approach as
> particularly elegant.  For example the webtest reports are not so clear
> (because of the grouping). But it does the job for now.  It would be
> interesting to see if there is interest in an improvement to this
> technique.
>
> The "soft" closure is defined as
>
>    def soft={ description, steps ->
>      groovy { log.info("Executing soft steps : $description ") }
>      retry (maxcount:2,counterName:'i') {
>        try {
>          ifStep {
>            condition { verifyProperty name:'i',value:0 }
>            then { steps() }
>          }
>          ifStep {
>            condition { not { verifyProperty name:'i',value:0 } }
>            then { groovy { softFailures << description }  }
>          }
>        } catch (e) { log.error(e) }
>      }
>    }
>
> (Side Note : I couldn't use the "else" step in groovy because it clashes
> with the groovy / java else syntax - anyone know if there is a way of
> addressing this conflict?)
>
> and "verfifyNoSoftFailures" method is defined as
>
>    def verfifyNoSoftFailures() {
>      softFailures.each { description ->
>        try {
>          webtest(description) {
>            storeProperty name:'fail',value:'false'
>            verifyProperty description:"$description
> (failed)",name:'fail',value:'true'
>          }
>        } catch (e) { log.error(e) }
>      }
>      softFailures.clear()
>    }
>
> (Side Note: I couldn't find a simple step to trigger a failure hence
> I've done the verbose storeProperty/verifyProperty to trigger a failure
> - anyone know how to to do this in a cleaner way?)
>
> and I have a class variable "softFailures" to store the soft failures
>
>   def softFailures=[]
>
> Cheers,
>
> Ian
>
> On 11 Jun 2009, at 19:27, John Spann wrote:
>
>> Hi Ian,
>>
>> This is actually very easy to implement, although we write xml tests
>> with ant macros, which I don't know how to re-use in groovy tests.
>> You should be able to utilize the same logic inside your groovy tests,
>> though.
>>
>> First, we wrote a macro which allows for a soft failure:
>>
>> <macrodef  name="softFail"  description="Wraps Steps such that the
>> failures are ignored">
>>    <element name="steps" implicit="true"/>
>>    <sequential>
>>        <retry description="softFail Wrapper" maxcount="2"
>> counterName="softFailCounter">
>>            <ifStep>
>>                <condition>
>>                    <verifyProperty description="Verify this is the
>> first run" name="softFailCounter" value="0" propertyType="dynamic" />
>>                </condition>
>>                <then>
>>                    <steps/>
>>                </then>
>>                <else>
>>                    <setProperty description="Set softFailure to true
>> so the test will fail" name="softFailure" value="true"
>> propertyType="dynamic" />
>>                </else>
>>            </ifStep>
>>        </retry>    </sequential> </macrodef>
>>
>> We then wrote a second macro which verifies no failure occurred:
>>
>> <macrodef name="checkSoftFailure" description="Fail the test if a soft
>> failure occurred">
>>    <sequential>
>>        <not>
>>            <verifyProperty description="Check if #{softFailure} is set
>> to true" name="softFailure" value="true" propertyType="dynamic" />
>>        </not>
>>    </sequential>
>> </macrodef>
>>
>> Our tests then look like this:
>>
>> <webtest name="Soft Fail Example">
>>    <steps>
>>        <!-- Do stuff -->
>>        <softFail>
>>                <!--Step or steps that can fail -->
>>        </softFail>
>>        <!-- Do more stuff -->
>>        <checkSoftFailure/>
>>    </steps>
>> </webtest>
>>
>> Hope that helps,
>>
>> John Spann | Associate Software Engineer
>>
>> Citrix Online Division
>> Citrix Systems, Inc.
>> 6500 Hollister Avenue
>> Goleta, CA 93117 USA
>> www.citrix.com
>>
>> Phone: 805.690.3489
>> Cell: 805.729.0008
>> Email: John.Spann@...
>>
>>
>> ________________________________
>> From: Ian Homer <ian.homer@...>
>> Reply-To: <webtest@...>, Ian Homer <ian.homer@...>
>> Date: Thu, 11 Jun 2009 01:17:48 -0700
>> To: <webtest@...>
>> Subject: [Webtest] RE: Continuing tests after a step failure
>>
>> Hi,
>>
>> Just reading back catalog of topics on the mailing list having
>> questioned to myself whether tests could continue a step failure and I
>> found this thread.
>>
>> Personally I think there is some value in having a soft step failure,
>> i.e. a step failure that does not cause subsequent steps in a webtest
>> to fail.  For example I have a test case that involves multiple forms
>> and I want to verify the title of each form as I go through the
>> flow.   I ran this test and it failed on the second form because the
>> title on the second page was not correct.  That's fine - because
>> that's a failed test, however this failure does not in reality prevent
>> the functional flow from continuing, the second form can still be
>> filled in and run OK.  Now I could out the tests into separate
>> webtests, and I could write the steps as macrodefs (or Groovy closures
>> as I'm writing tests in Groovy), but this seems an unnecessary
>> abstraction when I have a single script which I'd like to cover
>> various functional aspects.
>>
>> For example my groovy test script might be written as ...
>>
>>       webtest("test name") {
>>         group login
>>         verifyTitle 'Title 1'
>>         group inputForm1.curry(name:'Test',age:
>> 65,sex:'Female',smoker:'No')
>>         verifyTitle 'Title 2'
>>        group inputForm2.curry(name:'Test',age:65,sex:'Male',smoker:'No')
>>         verifyText 'Title 3'
>>       }
>>
>> (where login, inputForm1 and inputForm2 are closures defining how I
>> log in and fill in the two forms)
>>
>> Having a construct where I could identify one of the steps as a soft
>> failure, e.g.
>>
>>   soft { verifyTitle 'Title 2' }
>>
>> would be extremely useful, keep my test as a single script and allow
>> me to identify steps which do not break the flow on failure.
>>
>> What's the general consensus on this and would this approach be
>> something that would be considered?
>>
>> Ian
>>
>> --
>> Ian Homer
>> mobile ... made simple
>> http://bemoko.com | twitter: ianhomer
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> WebTest mailing list
>> WebTest@...
>> http://lists.canoo.com/mailman/listinfo/webtest
>>
>