Re: RE: Continuing tests after a step failure

>
> Some time ago (one or two years if I correctly remember) we already  
> had
> a similar discussion in this list. I think too that it could be
> interesting to have the possibility to "collect" verification errors  
> and
> make the test fail at the end. The xml report format needs to be
> enhanced for that to be able to store more than one failure per test.
> As always: a patch is welcome ;-)
>

> Cheers,
> Marc.
> --
> Web: http://www.efficient-webtesting.com
> Blog: http://mguillem.wordpress.com
>
> Ian Homer wrote:
>> Nice one ... I've taken your example into the groovy world with ...
>>
>>      def fullTestName="My Test"
>>      webtest() {
>>        group login
>>        group soft.curry("$fullTestName : Title 1", {verifyTitle  
>> 'Title
>> 1'})
>>        group inputForm1.curry(name:'Test',age:
>> 65,sex:'Female',smoker:'No')
>>        group soft.curry("$fullTestName : Title 2", {verifyTitle  
>> 'Title
>> 2'})
>>        group inputForm2.curry(name:'Test',age:
>> 65,sex:'Male',smoker:'No')
>>        group soft.curry("$fullTestName : Title 3", {verifyTitle  
>> 'Title
>> 3'})
>>      }
>>      verfifyNoSoftFailures()
>>
>> All soft failures are stored whilst running the main webtest and then
>> reported as failures in separate webtests.  I don't see this  
>> approach as
>> particularly elegant.  For example the webtest reports are not so  
>> clear
>> (because of the grouping). But it does the job for now.  It would be
>> interesting to see if there is interest in an improvement to this
>> technique.
>>
>> The "soft" closure is defined as
>>
>>   def soft={ description, steps ->
>>     groovy { log.info("Executing soft steps : $description ") }
>>     retry (maxcount:2,counterName:'i') {
>>       try {
>>         ifStep {
>>           condition { verifyProperty name:'i',value:0 }
>>           then { steps() }
>>         }
>>         ifStep {
>>           condition { not { verifyProperty name:'i',value:0 } }
>>           then { groovy { softFailures << description }  }
>>         }
>>       } catch (e) { log.error(e) }
>>     }
>>   }
>>
>> (Side Note : I couldn't use the "else" step in groovy because it  
>> clashes
>> with the groovy / java else syntax - anyone know if there is a way of
>> addressing this conflict?)
>>
>> and "verfifyNoSoftFailures" method is defined as
>>
>>   def verfifyNoSoftFailures() {
>>     softFailures.each { description ->
>>       try {
>>         webtest(description) {
>>           storeProperty name:'fail',value:'false'
>>           verifyProperty description:"$description
>> (failed)",name:'fail',value:'true'
>>         }
>>       } catch (e) { log.error(e) }
>>     }
>>     softFailures.clear()
>>   }
>>
>> (Side Note: I couldn't find a simple step to trigger a failure hence
>> I've done the verbose storeProperty/verifyProperty to trigger a  
>> failure
>> - anyone know how to to do this in a cleaner way?)
>>
>> and I have a class variable "softFailures" to store the soft failures
>>
>>  def softFailures=[]
>>
>> Cheers,
>>
>> Ian
>>
>> On 11 Jun 2009, at 19:27, John Spann wrote:
>>
>>> Hi Ian,
>>>
>>> This is actually very easy to implement, although we write xml tests
>>> with ant macros, which I don't know how to re-use in groovy tests.
>>> You should be able to utilize the same logic inside your groovy  
>>> tests,
>>> though.
>>>
>>> First, we wrote a macro which allows for a soft failure:
>>>
>>> <macrodef  name="softFail"  description="Wraps Steps such that the
>>> failures are ignored">
>>>   <element name="steps" implicit="true"/>
>>>   <sequential>
>>>       <retry description="softFail Wrapper" maxcount="2"
>>> counterName="softFailCounter">
>>>           <ifStep>
>>>               <condition>
>>>                   <verifyProperty description="Verify this is the
>>> first run" name="softFailCounter" value="0"  
>>> propertyType="dynamic" />
>>>               </condition>
>>>               <then>
>>>                   <steps/>
>>>               </then>
>>>               <else>
>>>                   <setProperty description="Set softFailure to true
>>> so the test will fail" name="softFailure" value="true"
>>> propertyType="dynamic" />
>>>               </else>
>>>           </ifStep>
>>>       </retry>    </sequential> </macrodef>
>>>
>>> We then wrote a second macro which verifies no failure occurred:
>>>
>>> <macrodef name="checkSoftFailure" description="Fail the test if a  
>>> soft
>>> failure occurred">
>>>   <sequential>
>>>       <not>
>>>           <verifyProperty description="Check if #{softFailure} is  
>>> set
>>> to true" name="softFailure" value="true" propertyType="dynamic" />
>>>       </not>
>>>   </sequential>
>>> </macrodef>
>>>
>>> Our tests then look like this:
>>>
>>> <webtest name="Soft Fail Example">
>>>   <steps>
>>>       <!-- Do stuff -->
>>>       <softFail>
>>>               <!--Step or steps that can fail -->
>>>       </softFail>
>>>       <!-- Do more stuff -->
>>>       <checkSoftFailure/>
>>>   </steps>
>>> </webtest>
>>>
>>> Hope that helps,
>>>
>>> John Spann | Associate Software Engineer
>>>
>>> Citrix Online Division
>>> Citrix Systems, Inc.
>>> 6500 Hollister Avenue
>>> Goleta, CA 93117 USA
>>> www.citrix.com
>>>
>>> Phone: 805.690.3489
>>> Cell: 805.729.0008
>>> Email: John.Spann@...
>>>
>>>
>>> ________________________________
>>> From: Ian Homer <ian.homer@...>
>>> Reply-To: <webtest@...>, Ian Homer <ian.homer@...>
>>> Date: Thu, 11 Jun 2009 01:17:48 -0700
>>> To: <webtest@...>
>>> Subject: [Webtest] RE: Continuing tests after a step failure
>>>
>>> Hi,
>>>
>>> Just reading back catalog of topics on the mailing list having
>>> questioned to myself whether tests could continue a step failure  
>>> and I
>>> found this thread.
>>>
>>> Personally I think there is some value in having a soft step  
>>> failure,
>>> i.e. a step failure that does not cause subsequent steps in a  
>>> webtest
>>> to fail.  For example I have a test case that involves multiple  
>>> forms
>>> and I want to verify the title of each form as I go through the
>>> flow.   I ran this test and it failed on the second form because the
>>> title on the second page was not correct.  That's fine - because
>>> that's a failed test, however this failure does not in reality  
>>> prevent
>>> the functional flow from continuing, the second form can still be
>>> filled in and run OK.  Now I could out the tests into separate
>>> webtests, and I could write the steps as macrodefs (or Groovy  
>>> closures
>>> as I'm writing tests in Groovy), but this seems an unnecessary
>>> abstraction when I have a single script which I'd like to cover
>>> various functional aspects.
>>>
>>> For example my groovy test script might be written as ...
>>>
>>>      webtest("test name") {
>>>        group login
>>>        verifyTitle 'Title 1'
>>>        group inputForm1.curry(name:'Test',age:
>>> 65,sex:'Female',smoker:'No')
>>>        verifyTitle 'Title 2'
>>>       group inputForm2.curry(name:'Test',age:
>>> 65,sex:'Male',smoker:'No')
>>>        verifyText 'Title 3'
>>>      }
>>>
>>> (where login, inputForm1 and inputForm2 are closures defining how I
>>> log in and fill in the two forms)
>>>
>>> Having a construct where I could identify one of the steps as a soft
>>> failure, e.g.
>>>
>>>  soft { verifyTitle 'Title 2' }
>>>
>>> would be extremely useful, keep my test as a single script and allow
>>> me to identify steps which do not break the flow on failure.
>>>
>>> What's the general consensus on this and would this approach be
>>> something that would be considered?
>>>
>>> Ian
>>>
>>> --
>>> Ian Homer
>>> mobile ... made simple
>>> http://bemoko.com | twitter: ianhomer
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> WebTest mailing list
>>> WebTest@...
>>> http://lists.canoo.com/mailman/listinfo/webtest
>>>
>>
>
> _______________________________________________
> WebTest mailing list
> WebTest@...
> http://lists.canoo.com/mailman/listinfo/webtest