Interaction Testing Automation

[Naoko Komura-2]

Thought to share the early concept of automating user interface testing
using Selenium.  The following plan was outlined by Ryan Lane.  The goal
is to have the central location of client testing, and open up the test
case submission  to MediaWiki authors and allows the reuse of test cases
simultaneously to multiple users.

http://usability.wikimedia.org/wiki/Resources#Interaction_testing_automation

Feel free to add your comments and input to the discussion page.  
(preferred over email thread)

Will keep you all posted with the progress.

Thanks,

- Naoko

Usability Initiative



--
Support Free Knowledge:  http://wikimediafoundation.org/wiki/Donate


_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Robert Rohde]

On Tue, Oct 27, 2009 at 9:58 AM, Naoko Komura <nkomura@...> wrote:

> Thought to share the early concept of automating user interface testing
> using Selenium.  The following plan was outlined by Ryan Lane.  The goal
> is to have the central location of client testing, and open up the test
> case submission  to MediaWiki authors and allows the reuse of test cases
> simultaneously to multiple users.
>
> http://usability.wikimedia.org/wiki/Resources#Interaction_testing_automation
>
> Feel free to add your comments and input to the discussion page.
> (preferred over email thread)
>
> Will keep you all posted with the progress.
>
> Thanks,
>
> - Naoko
>
> Usability Initiative

This sounds like a good idea, but I'm having difficulty telling from
the Selenium documentation what the output looks like or what it is
able to test.  It appears that one runs scripts in (or on top of) a
browser interface, so I assume the output is either the resulting HTML
after appropriate widgets have been clicked or an image of the
resulting rendered page?  And so one would then compare that content
to previous versions and other browser versions of the same page to
check for bugs and regressions, yes?  Is that gist of how Selenium is
designed to operate?

-Robert Rohde

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Marcus Davy]

Yes,
selenium can examine aspects of the html output as you run the tests.

Some time ago I worked in an extension in MediaWiki to store Selenium tests
in a test suite category and interact with Selenium via a special page.

http://www.mediawiki.org/wiki/Extension:Selenium

There is an example from links below, which runs the suite;

http://www.organicdesign.co.nz/wiki/extensions/Selenium/selenium-core/core/TestRunner.html?test=%2Fwiki%2Findex.php%3Ftitle%3DSpecial%3ASelenium%26suite%3DExample+Selenium+test+suite

http://www.organicdesign.co.nz/Example_Selenium_test_suite

Unfortunately the 'Create Account' test currently fails because of the
second layer of information required, but you can manually add that
information to more on to other tests such as editing an article etc.


Marcus


On Wed, Oct 28, 2009 at 9:50 AM, Robert Rohde <rarohde@...> wrote:

> On Tue, Oct 27, 2009 at 9:58 AM, Naoko Komura <nkomura@...>
> wrote:
> > Thought to share the early concept of automating user interface testing
> > using Selenium.  The following plan was outlined by Ryan Lane.  The goal
> > is to have the central location of client testing, and open up the test
> > case submission  to MediaWiki authors and allows the reuse of test cases
> > simultaneously to multiple users.
> >
> >
> http://usability.wikimedia.org/wiki/Resources#Interaction_testing_automation
> >
> > Feel free to add your comments and input to the discussion page.
> > (preferred over email thread)
> >
> > Will keep you all posted with the progress.
> >
> > Thanks,
> >
> > - Naoko
> >
> > Usability Initiative
>
> This sounds like a good idea, but I'm having difficulty telling from
> the Selenium documentation what the output looks like or what it is
> able to test.  It appears that one runs scripts in (or on top of) a
> browser interface, so I assume the output is either the resulting HTML
> after appropriate widgets have been clicked or an image of the
> resulting rendered page?  And so one would then compare that content
> to previous versions and other browser versions of the same page to
> check for bugs and regressions, yes?  Is that gist of how Selenium is
> designed to operate?
>
> -Robert Rohde
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@...
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[^demon]

On Tue, Oct 27, 2009 at 12:58 PM, Naoko Komura <nkomura@...> wrote:

> Thought to share the early concept of automating user interface testing
> using Selenium.  The following plan was outlined by Ryan Lane.  The goal
> is to have the central location of client testing, and open up the test
> case submission  to MediaWiki authors and allows the reuse of test cases
> simultaneously to multiple users.
>
> http://usability.wikimedia.org/wiki/Resources#Interaction_testing_automation
>
> Feel free to add your comments and input to the discussion page.
> (preferred over email thread)
>
> Will keep you all posted with the progress.
>
> Thanks,
>
> - Naoko
>
> Usability Initiative
>
>
>
> --
> Support Free Knowledge:  http://wikimediafoundation.org/wiki/Donate
>
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@...
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>

Didn't we just discuss this recently? :) Indeed yes, improving our test suite
is one of the perennial discussions on this list, along with rewriting the
parser and trying to properly sort non-English locales. We're very good at
beating the dead horse here. Couple of things that come to mind from the
last time we discussed this, so we don't have to rehash the obvious.

Test cases are no good if they're not used regularly. We've had some test cases
sitting in the Mediawiki repository since the beginning of time, and
they largely
haven't been touched since. They've been allowed to bitrot, to the
point of being
pretty much useless at this point. The problem is that they were never run
regularly (at all), so nobody cared whether they passed or not. The
rare exception
to this is the parser tests, which have been used very heavily since
their inception,
and are now integrated to Code Review--any commit to core code causes the
parser tests to be run to check for regressions.

If any new tests are to be successful, I think they should be _required_ to be
integrated to the code review process. Forced running of tests and clearly
displaying the results helps clearly track and identify when
regressions occurred,
so it's easier to fix them when they happen--which they do. I'd like to see a
whole lot of other tests created to cover many aspects of the code, and keeping
them as simple and straightforward as possible would be ideal. Making tests
easy to write makes developers more likely to write them to include with their
patches. If you make the learning curve too hard, developers won't bother, and
you've shot yourself in the foot :)

Just my 0.02USD

-Chad

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Ryan Lane-2]

> Test cases are no good if they're not used regularly. We've had some test cases
> sitting in the Mediawiki repository since the beginning of time, and
> they largely
> haven't been touched since. They've been allowed to bitrot, to the
> point of being
> pretty much useless at this point. The problem is that they were never run
> regularly (at all), so nobody cared whether they passed or not. The
> rare exception
> to this is the parser tests, which have been used very heavily since
> their inception,
> and are now integrated to Code Review--any commit to core code causes the
> parser tests to be run to check for regressions.
>
> If any new tests are to be successful, I think they should be _required_ to be
> integrated to the code review process. Forced running of tests and clearly
> displaying the results helps clearly track and identify when
> regressions occurred,
> so it's easier to fix them when they happen--which they do. I'd like to see a
> whole lot of other tests created to cover many aspects of the code, and keeping
> them as simple and straightforward as possible would be ideal. Making tests
> easy to write makes developers more likely to write them to include with their
> patches. If you make the learning curve too hard, developers won't bother, and
> you've shot yourself in the foot :)
>

There was also a comment on the talk page from Michael Dale mentioning
that this should integrate with Code Review. This is definitely a good
idea, and I'll try to work it into the plan.

As for the tests, simple tests seem fairly easy to create, as you can
use Selenium IDE to record actions, then output the script in a few
different languages (I'm assuming we'll use PHP). I think the hardest
part is going to be keeping the tests up to date with the code.

Respectfully,

Ryan Lane

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Aryeh Gregor]

On Tue, Oct 27, 2009 at 9:47 PM, Ryan Lane <rlane32@...> wrote:
> I think the hardest
> part is going to be keeping the tests up to date with the code.

That's pretty easy -- just have Code Review complain whenever anyone
causes a test failure, and force them to fix the tests they broke or
get reverted.  This assumes that it's easy to look at the changed
output and change the test so it's marked correct, though.  That seems
like it *should* be the case, at a glance, if the tests are
well-written.

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Ryan Lane-2]

> This sounds like a good idea, but I'm having difficulty telling from
> the Selenium documentation what the output looks like or what it is
> able to test.  It appears that one runs scripts in (or on top of) a
> browser interface, so I assume the output is either the resulting HTML
> after appropriate widgets have been clicked or an image of the
> resulting rendered page?  And so one would then compare that content
> to previous versions and other browser versions of the same page to
> check for bugs and regressions, yes?  Is that gist of how Selenium is
> designed to operate?
>

The scripts will run on a set of servers, using Selenium RC. Selenium
RC will run a browser on the server, and emulate a user doing actions.
It will return pass or fail on the conditions of your test.

For example, here is a php script that goes to the main page of
enwiki, types test into the search box, hits enter, then verifies that
the title is "Test - Wikipedia, the free encyclopedia". This test will
return as a pass.

<?php

require_once 'PHPUnit/Extensions/SeleniumTestCase.php';

class Example extends PHPUnit_Extensions_SeleniumTestCase
{
  function setUp()
  {
    $this->setBrowser("*chrome");
    $this->setBrowserUrl("http://en.wikipedia.org");
  }

  function testMyTestCase()
  {
    $this->open("/wiki/Main_Page");
    $this->type("searchInput", "test");
    $this->click("searchGoButton");
    $this->waitForPageToLoad("30000");
    $this->assertEquals("Test - Wikipedia, the free encyclopedia",
$this->getTitle());
  }
}
?>

Respectfully,

Ryan Lane

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[William Pietri]

Aryeh Gregor wrote:

> On Tue, Oct 27, 2009 at 9:47 PM, Ryan Lane <rlane32@...> wrote:
>  
>> I think the hardest
>> part is going to be keeping the tests up to date with the code.
>>    
>
> That's pretty easy -- just have Code Review complain whenever anyone
> causes a test failure, and force them to fix the tests they broke or
> get reverted.  This assumes that it's easy to look at the changed
> output and change the test so it's marked correct, though.  That seems
> like it *should* be the case, at a glance, if the tests are
> well-written.
>  

I'd encourage folks to start small with this.

One of the common failure modes of projects adopting automated
end-to-end tests (like Selenium) is that somebody divorced from the
development process makes a bunch of tests, often with playback and
record tools. As developers change things, they have a hard time
figuring out what tests mean or how to update them; that can either
reduce commits or encourage people to disable tests rather than updating
them.

I say this not to discourage the effort; I'm a huge fan of automated
tests. I just think that to maximize the chance of success, it's better
for the early focus to be on utility and sustainability rather than
maximum test coverage.

William

_______________________________________________
Wikitech-l mailing list
Wikitech-l@...
https://lists.wikimedia.org/mailman/listinfo/wikitech-l