Swing Automated Test Harness (SwATH)

[Eric Kolotyluk-2]

I have an idea for a new feature to be built into the Swing
Libraries and Java runtime.

One of the difficult areas of software testing is Graphical User
Interfaces. It is particularly hard to automate such testing because
of the dynamic and interactive nature of the GUI.

There are a number of tools on the market to aid in automated GUI
testing but these usually require a huge investment in design,
implementation, and (well) testing, and are often not very flexible
especially in regression testing when the UI changes.

I have some experience in automated testing in wireless networking.
Automating testing was very practical and straightforward because
usually you could create a test harness that simply monitored and
replied to network traffic and protocol simulation.

For some time now I have desired something similar for GUI testing.
What I think would work well is to embed a generic test harness into
the Swing API. The test harness would consist of a specified TCP
port to communication with the test system. For example run your
program with something like

-Xswath=123.123.123.123:1234

or some URL. All Swing activities would be translated into a series
of well defined XML messages. For example

<swath:component class="com.kodak.MyFrame" id="firstFrame">
  <swath:superclass class="javax.swing.JFrame">
    <swath:superclass class="java.awt.Frame">
      . . .
    </swath:superclass>
  </swath:superclass>
  <swath:component class="javax.swing.JButton" id="myButton">
    <swath:property name="isEnabled" type="boolean" value="true"/>
  </swath:component>
</swath:component>

This is a very crude example. The point is:

This XML would be emitted every time there is a change in the UI,
with little or no special effort on the part of the programmer. For
example the ids might have to be set by the programmer, but the rest
would be automatically generated similar to serialization. This XML
would be easily parsed by the automated test system using a variety
of technologies.

In turn the test system could respond with actions, for example

<swath:component id="myButton">
  <swath:action type="press"/>
</swath:component>


One important feature of such a system is that over time the UI
could change, the button might move to a different location, the
text on the button could be in a different language and font, an
icon could be added to the button, but the button's id remains the
same. This makes regression testing easier.

I feel there would be a great deal of utility in supporting such a
feature in Swing. Through automated testing of the GUI, the cost of
testing could be reduced dramatically, and the quality of software
could be assured more affordably.

There are a great many other issues and related opportunities for
specification and standardization. For example, tools to help build
the XML recognition engine in the test system; integration with
JUnit and other technologies; IDE integration and plugins; etc.

I would greatly appreciate hearing from someone on whether this
idea/proposal has merit.

[Carl Nagle]

I think you need to simply investigate the existence of several opensource or "free" testing tools that already support the very type of GUI testing you envision--some, like Abbot (http://abbot.sourceforge.net), are already plugged into the JUnit testing world.


Carl Nagle
SAS Institute http://www.sas.com
Project Architect, SAFSDEV
http://safsdev.sourceforge.net

[David Herron-3]

Interesting idea .. but...

The email address I'm using is not my work address @ sun.  As one of the
authors of the Robot class let me mention one of the policy decisions we
made at that time.  I don't know if the current AWT/Swing team feels this
way, but at the time we put in Robot we didn't want to make any
predetermined policy decisions about how testing would be accomplished.
For example we could have embedded into AWT/Swing something akin to Jemmy,
and by doing so it might have headed off the inventiveness in the public
who have gone on to develop alternative tools like MarathonMan or Abbot.
That's why Robot is so minimalized, we wanted to provide the bottom level
rudiments of test automation and allow the public to build on that in
whatever way they saw best.

For your idea ...

Maybe I don't understand the word "test harness", but to my eye what
you've described is more like Observability.  The ability to observe some
of the system innards.  To my understanding a "test harness" knows about a
set of test scenarios that can be run, and runs them one at a time in
sequence, and records the status of each scenario.  For a GUI test, the
scenario is a series of events that are squirted into the application
along with verification steps during the scenario execution that make sure
the application is behaving right.

I'm thinking there's probably already a way to construct an observability
system for Swing.  e.g. JMX or dtrace?

Also I once implemented an some code that would attach to every listener
method in every GUI component and print data on every execution of every
listener method.  Maybe you could write a similar thing which hooked up to
all bean properties of all GUI components in the application, and printed
data on every change of each bean property.

One level of validating the GUI is working properly is to verify the logic
is behaving right.  Are the right methods being called?  Do the component
values contain the right things?  That's what you'd be able to do by
tracking the component attributes like you describe.

Would this catch errors in the business logic of the application?

Would this catch rendering errors in the application?  Rendering errors
are particularly tricky to catch..AND.. are very expensive to test because
they so often must be manually verified.

- David Herron


On Wed, 3 Jan 2007, Eric Kolotyluk wrote:

> I have an idea for a new feature to be built into the Swing
> Libraries and Java runtime.
>
> One of the difficult areas of software testing is Graphical User
> Interfaces. It is particularly hard to automate such testing because
> of the dynamic and interactive nature of the GUI.
>
> There are a number of tools on the market to aid in automated GUI
> testing but these usually require a huge investment in design,
> implementation, and (well) testing, and are often not very flexible
> especially in regression testing when the UI changes.
>
> I have some experience in automated testing in wireless networking.
> Automating testing was very practical and straightforward because
> usually you could create a test harness that simply monitored and
> replied to network traffic and protocol simulation.
>
> For some time now I have desired something similar for GUI testing.
> What I think would work well is to embed a generic test harness into
> the Swing API. The test harness would consist of a specified TCP
> port to communication with the test system. For example run your
> program with something like
>
> -Xswath=123.123.123.123:1234
>
> or some URL. All Swing activities would be translated into a series
> of well defined XML messages. For example
>
> <swath:component class="com.kodak.MyFrame" id="firstFrame">
>   <swath:superclass class="javax.swing.JFrame">
>     <swath:superclass class="java.awt.Frame">
>       . . .
>     </swath:superclass>
>   </swath:superclass>
>   <swath:component class="javax.swing.JButton" id="myButton">
>     <swath:property name="isEnabled" type="boolean" value="true"/>
>   </swath:component>
> </swath:component>
>
> This is a very crude example. The point is:
>
> This XML would be emitted every time there is a change in the UI,
> with little or no special effort on the part of the programmer. For
> example the ids might have to be set by the programmer, but the rest
> would be automatically generated similar to serialization. This XML
> would be easily parsed by the automated test system using a variety
> of technologies.
>
> In turn the test system could respond with actions, for example
>
> <swath:component id="myButton">
>   <swath:action type="press"/>
> </swath:component>
>
>
> One important feature of such a system is that over time the UI
> could change, the button might move to a different location, the
> text on the button could be in a different language and font, an
> icon could be added to the button, but the button's id remains the
> same. This makes regression testing easier.
>
> I feel there would be a great deal of utility in supporting such a
> feature in Swing. Through automated testing of the GUI, the cost of
> testing could be reduced dramatically, and the quality of software
> could be assured more affordably.
>
> There are a great many other issues and related opportunities for
> specification and standardization. For example, tools to help build
> the XML recognition engine in the test system; integration with
> JUnit and other technologies; IDE integration and plugins; etc.
>
> I would greatly appreciate hearing from someone on whether this
> idea/proposal has merit.
>
>
>

[shura-4]

Hi, Eric.

If I got what you're saying ...

What you're trying to achieve is discoverability of UI components  
through the product live cycle. The component ID is what your test  
relies on. The id value is then supported by development.

I have to point out that there is an instrument which could be used  
for that right now. To the best of my knowledge, Component.getName()  
method (or "name" property, if you wish) is what was created exactly  
for that point - to store some kind of component identifier which is  
not used for anything else but the purpose of identifying the  
components. Which a little of support from test tools, using of  
"name" property would be just the same useful as your proposal.

On the other hand, though, component discoverability is not a  
panacea. There is no point in keeping the IDs intact, if components  
could disappear completely. Check an example used in this article  
(http://jemmy.netbeans.org/COT.html, particularly, "UI-dependent  
library" chapter) - you will see what I mean.

Some discussions have happened in the past on this forum as to  
whether it is possible to store application logic (not just  
components identification logic, but something much bigger) within  
the application itself. I personally think that deserves to be a  
fundamental principle of software development process - not just a  
testing approach. As such, it kinda lays outside of the scope of this  
forum.

Shura.



On Jan 3, 2007, at 9:48 PM, Eric Kolotyluk wrote:

> I have an idea for a new feature to be built into the Swing
> Libraries and Java runtime.
>
> One of the difficult areas of software testing is Graphical User
> Interfaces. It is particularly hard to automate such testing because
> of the dynamic and interactive nature of the GUI.
>
> There are a number of tools on the market to aid in automated GUI
> testing but these usually require a huge investment in design,
> implementation, and (well) testing, and are often not very flexible
> especially in regression testing when the UI changes.
>
> I have some experience in automated testing in wireless networking.
> Automating testing was very practical and straightforward because
> usually you could create a test harness that simply monitored and
> replied to network traffic and protocol simulation.
>
> For some time now I have desired something similar for GUI testing.
> What I think would work well is to embed a generic test harness into
> the Swing API. The test harness would consist of a specified TCP
> port to communication with the test system. For example run your
> program with something like
>
> -Xswath=123.123.123.123:1234
>
> or some URL. All Swing activities would be translated into a series
> of well defined XML messages. For example
>
> <swath:component class="com.kodak.MyFrame" id="firstFrame">
> <swath:superclass class="javax.swing.JFrame">
> <swath:superclass class="java.awt.Frame">
> . . .
> </swath:superclass>
> </swath:superclass>
> <swath:component class="javax.swing.JButton" id="myButton">
> <swath:property name="isEnabled" type="boolean" value="true"/>
> </swath:component>
> </swath:component>
>
> This is a very crude example. The point is:
>
> This XML would be emitted every time there is a change in the UI,
> with little or no special effort on the part of the programmer. For
> example the ids might have to be set by the programmer, but the rest
> would be automatically generated similar to serialization. This XML
> would be easily parsed by the automated test system using a variety
> of technologies.
>
> In turn the test system could respond with actions, for example
>
> <swath:component id="myButton">
> <swath:action type="press"/>
> </swath:component>
>
> One important feature of such a system is that over time the UI
> could change, the button might move to a different location, the
> text on the button could be in a different language and font, an
> icon could be added to the button, but the button's id remains the
> same. This makes regression testing easier.
>
> I feel there would be a great deal of utility in supporting such a
> feature in Swing. Through automated testing of the GUI, the cost of
> testing could be reduced dramatically, and the quality of software
> could be assured more affordably.
>
> There are a great many other issues and related opportunities for
> specification and standardization. For example, tools to help build
> the XML recognition engine in the test system; integration with
> JUnit and other technologies; IDE integration and plugins; etc.
>
> I would greatly appreciate hearing from someone on whether this
> idea/proposal has merit.
>
>
>

[Jiri Skrivanek]

I think everything you described is already possible. You can listen for
every event using Toolkit.getDefaultToolkit().addAWTEventListener().
Also you can use java.awt.Component.setName/getName to uniquely identify
a component and later find it in your test. Of course you have to run
your test within the same JVM as your application is running.
You are right that changes of UI during application development are
problematic. But I think that majority of UI testing libraries can
handle small layout changes (position, size). In my experience any UI
testing tool cannot overcome bigger UI changes. You have to modify your
test. It decreases effectivity of UI test automation and that's why is
better to write tests after the UI is somehow stabilized or even frozen.
BTW, I am using Jemmy (http://jemmy.netbeans.org/).

Jiri


Eric Kolotyluk wrote:

> I have an idea for a new feature to be built into the Swing
> Libraries and Java runtime.
>
> One of the difficult areas of software testing is Graphical User
> Interfaces. It is particularly hard to automate such testing because
> of the dynamic and interactive nature of the GUI.
>
> There are a number of tools on the market to aid in automated GUI
> testing but these usually require a huge investment in design,
> implementation, and (well) testing, and are often not very flexible
> especially in regression testing when the UI changes.
>
> I have some experience in automated testing in wireless networking.
> Automating testing was very practical and straightforward because
> usually you could create a test harness that simply monitored and
> replied to network traffic and protocol simulation.
>
> For some time now I have desired something similar for GUI testing.
> What I think would work well is to embed a generic test harness into
> the Swing API. The test harness would consist of a specified TCP
> port to communication with the test system. For example run your
> program with something like
>
> -Xswath=123.123.123.123:1234
>
> or some URL. All Swing activities would be translated into a series
> of well defined XML messages. For example
>
> <swath:component class="com.kodak.MyFrame" id="firstFrame">
>   <swath:superclass class="javax.swing.JFrame">
>     <swath:superclass class="java.awt.Frame">
>       . . .
>     </swath:superclass>
>   </swath:superclass>
>   <swath:component class="javax.swing.JButton" id="myButton">
>     <swath:property name="isEnabled" type="boolean" value="true"/>
>   </swath:component>
> </swath:component>
>
> This is a very crude example. The point is:
>
> This XML would be emitted every time there is a change in the UI,
> with little or no special effort on the part of the programmer. For
> example the ids might have to be set by the programmer, but the rest
> would be automatically generated similar to serialization. This XML
> would be easily parsed by the automated test system using a variety
> of technologies.
>
> In turn the test system could respond with actions, for example
>
> <swath:component id="myButton">
>   <swath:action type="press"/>
> </swath:component>
>
>
> One important feature of such a system is that over time the UI
> could change, the button might move to a different location, the
> text on the button could be in a different language and font, an
> icon could be added to the button, but the button's id remains the
> same. This makes regression testing easier.
>
> I feel there would be a great deal of utility in supporting such a
> feature in Swing. Through automated testing of the GUI, the cost of
> testing could be reduced dramatically, and the quality of software
> could be assured more affordably.
>
> There are a great many other issues and related opportunities for
> specification and standardization. For example, tools to help build
> the XML recognition engine in the test system; integration with
> JUnit and other technologies; IDE integration and plugins; etc.
>
> I would greatly appreciate hearing from someone on whether this
> idea/proposal has merit.
>
>
>

[Eric Kolotyluk-2]

--- In java-gui-testing@..., David Herron <davidh@...>
wrote:
>
>
> Interesting idea .. but...
>
> The email address I'm using is not my work address @ sun.  As one
of the
> authors of the Robot class let me mention one of the policy
decisions we
> made at that time.  I don't know if the current AWT/Swing team
feels this
> way, but at the time we put in Robot we didn't want to make any
> predetermined policy decisions about how testing would be
accomplished.
> For example we could have embedded into AWT/Swing something akin
to Jemmy,
> and by doing so it might have headed off the inventiveness in the
public
> who have gone on to develop alternative tools like MarathonMan or
Abbot.
> That's why Robot is so minimalized, we wanted to provide the
bottom level
> rudiments of test automation and allow the public to build on that
in
> whatever way they saw best.

That's a very good principle to follow. I think I need to become
more familiar with the Robot class. At first glance it does not
offer the higher level of UI stimulus I'm looking for (i.e. button
press), but as you said, it's something to build upon.

>
> For your idea ...
>
> Maybe I don't understand the word "test harness", but to my eye
what
> you've described is more like Observability.  The ability to
observe some
> of the system innards.  To my understanding a "test harness" knows
about a
> set of test scenarios that can be run, and runs them one at a time
in
> sequence, and records the status of each scenario.  For a GUI
test, the
> scenario is a series of events that are squirted into the
application
> along with verification steps during the scenario execution that
make sure
> the application is behaving right.
>
> I'm thinking there's probably already a way to construct an
observability
> system for Swing.  e.g. JMX or dtrace?

When I was in a test team at Motorola we referred to the thing that
ran the automated test cases/suites as the "Test System" and
the "Test Harness" was the piece between the Test System and the
Unit Under Test. In fact I would say observability is a very
important part of a Test Harness, as well as stimulating Unit Under
Test. For example, I would view Costello as the Test System and
Abbot as the Test Harness. Anyway, different places may use similar
terms differently, but that why I saw SwATH as a "Harness".

In particular (to me), the Test Harness simulates the external
environment of the Unit Under Test, and the Test System drives the
Unit Under Test via the Test Harness.

>
> Also I once implemented an some code that would attach to every
listener
> method in every GUI component and print data on every execution of
every
> listener method.  Maybe you could write a similar thing which
hooked up to
> all bean properties of all GUI components in the application, and
printed
> data on every change of each bean property.

I don't want to have to go to that much work. I don't know that it's
enough to attach to every listener. Also, another important feature
I want is to be able stimulate the UI at a higher level than the
Robot class allows.

>
> One level of validating the GUI is working properly is to verify
the logic
> is behaving right.  Are the right methods being called?  Do the
component
> values contain the right things?  That's what you'd be able to do
by
> tracking the component attributes like you describe.

SwATH wouldn't directly be able to test if the right methods are
being called, only that the UI contained the right values and
exhibited the correct behavior.

>
> Would this catch errors in the business logic of the application?

Yes, that is mainly what the intent is.

I only just now became aware of Abbot and Costello which seems very
similar to what I want (but in a very different way than I
imagined). However, I'd have to spend more time with them to
determine if they do what I really want. If they don't I'll revisit
the SwATH idea or try to discuss features I'd like to see in other
test systems.

One advantage of SwATH over Costello is that the Test System could
run on a different system. It looks to me like Abbot and Costello
need to run on the same system as the Unit Under Test. I'm also
concerned about how labor intensive it is to setup and use Abbot and
Costello.

Up until now all of our GUI Unit Testing has been manual. I'd like
to better understand the cost trade-offs between manual testing of
the GUI and writing test cases in Abbot, and test suites in Costello.

>
> Would this catch rendering errors in the application?  Rendering
errors
> are particularly tricky to catch..AND.. are very expensive to test
because
> they so often must be manually verified.

No, SwATH would definately not catch rendering errors. That is a
whole other class of problem that products like Redstone's Eggplant
are designed to deal with. I ruled out using Eggplant because it
looks too labor intensive and does not look resilient enough to
changes in the UI.

I think there will always be some level of testing that just
requires people to perform.

>
> - David Herron
>

Anyway, I really appreciate the feedback, it's given me a lot of
useful things to think about and go and explore.

Cheers, Eric

[Eric Kolotyluk-2]

Yes, definately. Thanks for the reference.

--- In java-gui-testing@..., "Carl Nagle" <Carl.Nagle@...>
wrote:
>
> I think you need to simply investigate the existence of several
opensource or "free" testing tools that already support the very type
of GUI testing you envision--some, like Abbot
(http://abbot.sourceforge.net), are already plugged into the JUnit
testing world.
>
>
> Carl Nagle
> SAS Institute http://www.sas.com
> Project Architect, SAFSDEV
> http://safsdev.sourceforge.net
>

[Eric Kolotyluk-2]

--- In java-gui-testing@..., Shura <alexandre.iline@...>
wrote:
>
> Hi, Eric.
>
> If I got what you're saying ...
>
> What you're trying to achieve is discoverability of UI components  
> through the product live cycle. The component ID is what your
test  
> relies on. The id value is then supported by development.

Yes, definately.

>
> I have to point out that there is an instrument which could be
used  
> for that right now. To the best of my knowledge, Component.getName
()  
> method (or "name" property, if you wish) is what was created
exactly  
> for that point - to store some kind of component identifier which
is  
> not used for anything else but the purpose of identifying the  
> components. Which a little of support from test tools, using of  
> "name" property would be just the same useful as your proposal.

Yes, that is exactly what I was thinking, and I should have said so
initially.

>
> On the other hand, though, component discoverability is not a  
> panacea. There is no point in keeping the IDs intact, if
components  
> could disappear completely. Check an example used in this article  
> (http://jemmy.netbeans.org/COT.html, particularly, "UI-dependent  
> library" chapter) - you will see what I mean.

I'm thinking of the UI-dependent library, and not the Concept-
dependent approach. I wish the article said more about the Concept-
dependent approach.

>
> Some discussions have happened in the past on this forum as to  
> whether it is possible to store application logic (not just  
> components identification logic, but something much bigger)
within  
> the application itself. I personally think that deserves to be a  
> fundamental principle of software development process - not just
a  
> testing approach. As such, it kinda lays outside of the scope of
this  
> forum.

No, I wasn't thinking of that, but it is an interesting idea.

>
> Shura.
>

Thanks for the feedback.

Cheers, Eric

[Eric Kolotyluk-2]

Thanks for the reference. In addition to Abbot and Costello, Jemmy
is one more test environment for me to check out.

Cheers, Eric

--- In java-gui-testing@..., Jiri Skrivanek
<Jiri.Skrivanek@...> wrote:
>
> I think everything you described is already possible. You can
listen for
> every event using Toolkit.getDefaultToolkit().addAWTEventListener
().
> Also you can use java.awt.Component.setName/getName to uniquely
identify
> a component and later find it in your test. Of course you have to
run
> your test within the same JVM as your application is running.
> You are right that changes of UI during application development
are
> problematic. But I think that majority of UI testing libraries can
> handle small layout changes (position, size). In my experience any
UI
> testing tool cannot overcome bigger UI changes. You have to modify
your
> test. It decreases effectivity of UI test automation and that's
why is
> better to write tests after the UI is somehow stabilized or even
frozen.
> BTW, I am using Jemmy (http://jemmy.netbeans.org/).
>
> Jiri
>

[finbarr_brady]

Hi Eric,

Have you looked into using 'Squish for Java' from Froglogic? It works
like Abbot, but is much more mature as a product. I have just begun
using it here in Ericsson and it is a great tool. It is the only tool
that is better than XRunner for us (on Solaris - works on all
platforms though). You should certainly have a look... (email the
company for an evaluation license)

http://www.froglogic.com/pg?id=Products&category=squish&sub=editions&subsub=java



--- In java-gui-testing@..., "Eric Kolotyluk" <eric@...>
wrote:

>
> --- In java-gui-testing@..., David Herron <davidh@>
> wrote:
> >
> >
> > Interesting idea .. but...
> >
> > The email address I'm using is not my work address @ sun.  As one
> of the
> > authors of the Robot class let me mention one of the policy
> decisions we
> > made at that time.  I don't know if the current AWT/Swing team
> feels this
> > way, but at the time we put in Robot we didn't want to make any
> > predetermined policy decisions about how testing would be
> accomplished.
> > For example we could have embedded into AWT/Swing something akin
> to Jemmy,
> > and by doing so it might have headed off the inventiveness in the
> public
> > who have gone on to develop alternative tools like MarathonMan or
> Abbot.
> > That's why Robot is so minimalized, we wanted to provide the
> bottom level
> > rudiments of test automation and allow the public to build on that
> in
> > whatever way they saw best.
>
> That's a very good principle to follow. I think I need to become
> more familiar with the Robot class. At first glance it does not
> offer the higher level of UI stimulus I'm looking for (i.e. button
> press), but as you said, it's something to build upon.
>
> >
> > For your idea ...
> >
> > Maybe I don't understand the word "test harness", but to my eye
> what
> > you've described is more like Observability.  The ability to
> observe some
> > of the system innards.  To my understanding a "test harness" knows
> about a
> > set of test scenarios that can be run, and runs them one at a time
> in
> > sequence, and records the status of each scenario.  For a GUI
> test, the
> > scenario is a series of events that are squirted into the
> application
> > along with verification steps during the scenario execution that
> make sure
> > the application is behaving right.
> >
> > I'm thinking there's probably already a way to construct an
> observability
> > system for Swing.  e.g. JMX or dtrace?
>
> When I was in a test team at Motorola we referred to the thing that
> ran the automated test cases/suites as the "Test System" and
> the "Test Harness" was the piece between the Test System and the
> Unit Under Test. In fact I would say observability is a very
> important part of a Test Harness, as well as stimulating Unit Under
> Test. For example, I would view Costello as the Test System and
> Abbot as the Test Harness. Anyway, different places may use similar
> terms differently, but that why I saw SwATH as a "Harness".
>
> In particular (to me), the Test Harness simulates the external
> environment of the Unit Under Test, and the Test System drives the
> Unit Under Test via the Test Harness.
>
> >
> > Also I once implemented an some code that would attach to every
> listener
> > method in every GUI component and print data on every execution of
> every
> > listener method.  Maybe you could write a similar thing which
> hooked up to
> > all bean properties of all GUI components in the application, and
> printed
> > data on every change of each bean property.
>
> I don't want to have to go to that much work. I don't know that it's
> enough to attach to every listener. Also, another important feature
> I want is to be able stimulate the UI at a higher level than the
> Robot class allows.
>
> >
> > One level of validating the GUI is working properly is to verify
> the logic
> > is behaving right.  Are the right methods being called?  Do the
> component
> > values contain the right things?  That's what you'd be able to do
> by
> > tracking the component attributes like you describe.
>
> SwATH wouldn't directly be able to test if the right methods are
> being called, only that the UI contained the right values and
> exhibited the correct behavior.
>
> >
> > Would this catch errors in the business logic of the application?
>
> Yes, that is mainly what the intent is.
>
> I only just now became aware of Abbot and Costello which seems very
> similar to what I want (but in a very different way than I
> imagined). However, I'd have to spend more time with them to
> determine if they do what I really want. If they don't I'll revisit
> the SwATH idea or try to discuss features I'd like to see in other
> test systems.
>
> One advantage of SwATH over Costello is that the Test System could
> run on a different system. It looks to me like Abbot and Costello
> need to run on the same system as the Unit Under Test. I'm also
> concerned about how labor intensive it is to setup and use Abbot and
> Costello.
>
> Up until now all of our GUI Unit Testing has been manual. I'd like
> to better understand the cost trade-offs between manual testing of
> the GUI and writing test cases in Abbot, and test suites in Costello.
>
> >
> > Would this catch rendering errors in the application?  Rendering
> errors
> > are particularly tricky to catch..AND.. are very expensive to test
> because
> > they so often must be manually verified.
>
> No, SwATH would definately not catch rendering errors. That is a
> whole other class of problem that products like Redstone's Eggplant
> are designed to deal with. I ruled out using Eggplant because it
> looks too labor intensive and does not look resilient enough to
> changes in the UI.
>
> I think there will always be some level of testing that just
> requires people to perform.
>
> >
> > - David Herron
> >
>
> Anyway, I really appreciate the feedback, it's given me a lot of
> useful things to think about and go and explore.
>
> Cheers, Eric
>