Testrunner option for running tests in random order?!?

[Christian Theune-2]

Hi,

I *thought* the test runner had a feature to run tests in randomized
order, but I can't find it. Is that something that Zope 2 had and that
was never ported to zope.testing?

I couldn't find in the history either.

/me is confused.

--
Christian Theune · ct@...
gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany
http://gocept.com · tel [phone] · fax [phone]
Zope and Plone consulting and development
_______________________________________________
Zope-Dev maillist  -  Zope-Dev@...
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )

[Jonathan Ballet-5]

Hi Christian,

On Fri, Oct 2, 2009 at 2:00 PM, Christian Theune <ct@...> wrote:
> Hi,
>
> I *thought* the test runner had a feature to run tests in randomized
> order, but I can't find it. Is that something that Zope 2 had and that
> was never ported to zope.testing?
>
> I couldn't find in the history either.
>
> /me is confused.

I was about to send the same mail today, but I found out your mail before.

I didn't find anything relative to random tests in Zope 2 (but I'm not
familiar at all with Zope 2, so I'm pretty sure I could have missed
something), and I ended up writing a new feature for zope.testing's
testrunner.

I made a branch 'shuffle-tests', available on the Mercurial repository
at http://bitbucket.org/multani/zope.testing-shuffle/ , which adds two
new options:
  * --shuffle to randomize the order of the tests in a layer (layers
are not yet randomized, I didn't look at it yet);
  * --shuffle-seed XXX, to initialize the RNG with a specific value,
which allows to reproduce a specific randomization of tests (inspired
by this thread http://lists.idyll.org/pipermail/testing-in-python/2009-April/001812.html
)

It's not yet ready-to-commit: it still misses tests (tricky to write)
and more documentation.
However, feedbacks are welcome!

Cheers,

 Jonathan
_______________________________________________
Zope-Dev maillist  -  Zope-Dev@...
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )

[Christian Theune-2]

Hi,

On 10/08/2009 03:28 PM, Jonathan Ballet wrote:

> Hi Christian,
>
> On Fri, Oct 2, 2009 at 2:00 PM, Christian Theune <ct@...> wrote:
>> Hi,
>>
>> I *thought* the test runner had a feature to run tests in randomized
>> order, but I can't find it. Is that something that Zope 2 had and that
>> was never ported to zope.testing?
>>
>> I couldn't find in the history either.
>>
>> /me is confused.
>
> I was about to send the same mail today, but I found out your mail before.
>
> I didn't find anything relative to random tests in Zope 2 (but I'm not
> familiar at all with Zope 2, so I'm pretty sure I could have missed
> something), and I ended up writing a new feature for zope.testing's
> testrunner.
>
> I made a branch 'shuffle-tests', available on the Mercurial repository
> at http://bitbucket.org/multani/zope.testing-shuffle/ , which adds two
> new options:
>
>   * --shuffle to randomize the order of the tests in a layer (layers
> are not yet randomized, I didn't look at it yet);
>   * --shuffle-seed XXX, to initialize the RNG with a specific value,
> which allows to reproduce a specific randomization of tests (inspired
> by this thread http://lists.idyll.org/pipermail/testing-in-python/2009-April/001812.html
> )

Sounds right. It could even be folded into a single option:

--shuffle[=XXX]

> It's not yet ready-to-commit: it still misses tests (tricky to write)
> and more documentation.

That's what working branches are for. You do have commit rights, don't you?

It's not that tricky to test: seed the RNG from outside the tests.

> However, feedbacks are welcome!

Thanks for the work!

Christian

--
Christian Theune · ct@...
gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany
http://gocept.com · tel [phone] · fax [phone]
Zope and Plone consulting and development

_______________________________________________
Zope-Dev maillist  -  Zope-Dev@...
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )

[Jonathan Ballet-5]

Hey,

On Fri, Oct 9, 2009 at 2:33 PM, Christian Theune <ct@...> wrote:

>>   * --shuffle to randomize the order of the tests in a layer (layers
>> are not yet randomized, I didn't look at it yet);
>>   * --shuffle-seed XXX, to initialize the RNG with a specific value,
>> which allows to reproduce a specific randomization of tests (inspired
>> by this thread http://lists.idyll.org/pipermail/testing-in-python/2009-April/001812.html
>> )
>
> Sounds right. It could even be folded into a single option:
>
> --shuffle[=XXX]

I'm not sure how to do that, considering those use cases:

 * the user doesn't want to shuffle its tests and so, doesn't set --shuffle
 * the user want to shuffle its tests, without specifying an order,
and use --shuffle
 * the user want to shuffle its tests in specific order, and use --shuffle=42

If I want this option to have an optional value, I think I must set a
default value to the option. But then, how to make the distinction
between the two first cases? I might miss something I don't know in
option parsing, but I didn't find anything...

>> It's not yet ready-to-commit: it still misses tests (tricky to write)
>> and more documentation.
>
> That's what working branches are for. You do have commit rights, don't you?

I don't (personally) have commit rights, but my boss (sdouche) has.
I'll see with him.


Thanks for your advices!

 Jonathan
_______________________________________________
Zope-Dev maillist  -  Zope-Dev@...
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )

[Benji York-2]

On Fri, Oct 9, 2009 at 8:51 AM, Jonathan Ballet
<jonathan.ballet@...> wrote:
> If I want this option to have an optional value, I think I must set a
> default value to the option. But then, how to make the distinction
> between the two first cases? I might miss something I don't know in
> option parsing, but I didn't find anything...

I'd try the same approach as default options in Python functions.  Make
the default None (which isn't a valid seed) and then you can tell the
difference.
--
Benji York
Senior Software Engineer
Zope Corporation
_______________________________________________
Zope-Dev maillist  -  Zope-Dev@...
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )

[Jonathan Ballet-5]

Hi,

On Fri, Oct 9, 2009 at 3:30 PM, Benji York <benji@...> wrote:

> On Fri, Oct 9, 2009 at 8:51 AM, Jonathan Ballet
> <jonathan.ballet@...> wrote:
>> If I want this option to have an optional value, I think I must set a
>> default value to the option. But then, how to make the distinction
>> between the two first cases? I might miss something I don't know in
>> option parsing, but I didn't find anything...
>
> I'd try the same approach as default options in Python functions.  Make
> the default None (which isn't a valid seed) and then you can tell the
> difference.

That's what I originally though, but then, how can I make the
difference between 'the user has not set --shuffle' (the option's
value is then None), and 'the user has set the --shuffle option, but
without specifying a seed' (then the option's value is None again)?

 Jonathan
_______________________________________________
Zope-Dev maillist  -  Zope-Dev@...
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )

[Marius Gedminas-4]

On Fri, Oct 09, 2009 at 02:33:02PM +0200, Christian Theune wrote:

> On 10/08/2009 03:28 PM, Jonathan Ballet wrote:
> > I made a branch 'shuffle-tests', available on the Mercurial repository
> > at http://bitbucket.org/multani/zope.testing-shuffle/ , which adds two
> > new options:
> >
> >   * --shuffle to randomize the order of the tests in a layer (layers
> > are not yet randomized, I didn't look at it yet);
> >   * --shuffle-seed XXX, to initialize the RNG with a specific value,
> > which allows to reproduce a specific randomization of tests (inspired
> > by this thread http://lists.idyll.org/pipermail/testing-in-python/2009-April/001812.html
> > )
>
> Sounds right. It could even be folded into a single option:
>
> --shuffle[=XXX]

I doubt that.  Ordinarily with the GNU long option style these two are
equivalent:

   --shuffle=42
   --shuffle 42

but there's no way to tell the second variant from a seed-less --shuffle
followed by an non-option argument.

Marius Gedminas
--
http://pov.lt/ -- Zope 3 consulting and development


_______________________________________________
Zope-Dev maillist  -  Zope-Dev@...
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )

[Sebastien Douche]

On Fri, Oct 9, 2009 at 14:33, Christian Theune <ct@...> wrote:

Hi Christian.

> That's what working branches are for.

Done :
http://svn.zope.org/zope.testing/branches/sdouche-shuffle/



--
Sebastien Douche <sdouche@...>
Twitter: http://bit.ly/afkrK (agile, python, open source)
_______________________________________________
Zope-Dev maillist  -  Zope-Dev@...
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )

[Jonathan Ballet-5]

Hello there,

as I mentioned by Sebastien, the patch has been commited into the
sdouche-shuffle branch. It now has tests and a bit of documentation
too.

Regarding CLI arguments, I'm not sure this could be changed like
Christian proposed. If you know how to do it, I'm listening, 'cause I
don't know! :)

I have one more thing I would like to discuss: how should this feature
behave with the --repeat feature?
Currently, the shuffling is done before the repeat feature. This means
that tests will *not* get reordered for each loop (just once, at the
very beginning of tests launching).
I know how to do it, so that for each loop, tests will get a different
order (basically, I would propose to wrap the __iter__() method of the
TestSuite used to launch tests, and to shuffle here):

    class ShuffledTestSuite(unittest.TestSuite):
        def __iter__(self):
            tests = list(super(Shuffled, self))
            seed = long(time.time() * 256)
            shuffler = random.Random()
            shuffler.shuffle(tests)
            return iter(tests)

However, using this means we would loose the seed printing (which
seems very convenient to me). I'm not sure if we could get it and
print it for each loop...


Reviews, comments and feedbacks are welcome!
Cheers,

 Jonathan
_______________________________________________
Zope-Dev maillist  -  Zope-Dev@...
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )

[Christian Theune-2]

On 10/19/2009 11:57 AM, Jonathan Ballet wrote:

> Hello there,
>
> as I mentioned by Sebastien, the patch has been commited into the
> sdouche-shuffle branch. It now has tests and a bit of documentation
> too.
>
> Regarding CLI arguments, I'm not sure this could be changed like
> Christian proposed. If you know how to do it, I'm listening, 'cause I
> don't know! :)
>
> I have one more thing I would like to discuss: how should this feature
> behave with the --repeat feature?
> Currently, the shuffling is done before the repeat feature. This means
> that tests will *not* get reordered for each loop (just once, at the
> very beginning of tests launching).
> I know how to do it, so that for each loop, tests will get a different
> order (basically, I would propose to wrap the __iter__() method of the
> TestSuite used to launch tests, and to shuffle here):
>
>     class ShuffledTestSuite(unittest.TestSuite):
>         def __iter__(self):
>             tests = list(super(Shuffled, self))
>             seed = long(time.time() * 256)
>             shuffler = random.Random()
>             shuffler.shuffle(tests)
>             return iter(tests)
>
> However, using this means we would loose the seed printing (which
> seems very convenient to me). I'm not sure if we could get it and
> print it for each loop...

I'd say leave it as is for now. I think it's more important to be able
to seed (for the sake of reproduction, so btw: do you log the seed if
you generated one?) externally than to reorder during each repeated run.

Also, we can add that later on in case you figure out a way how to do it. ;)

Christia

--
Christian Theune · ct@...
gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany
http://gocept.com · tel [phone] · fax [phone]
Zope and Plone consulting and development

_______________________________________________
Zope-Dev maillist  -  Zope-Dev@...
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )

[Jonathan Ballet-5]

Hello Christian,

On Tue, Nov 3, 2009 at 11:07 AM, Christian Theune <ct@...> wrote:
> [...] btw: do you log the seed if
> you generated one?)

Yep, the seed is displayed at the end of tests, so that it can be
reused to reproduce a failing test.

    $ ./bin/test --shuffle -cvvv
    Running tests at level 1
    Running zope.testing.testrunner.layer.UnitTests tests:
      Set up zope.testing.testrunner.layer.UnitTests in 0.000 seconds.
      ...
      Ran 257 tests with 0 failures and 0 errors in 21.908 seconds.
    Tests were shuffled using seed number 321854366611

I already find it *very* useful ...

Cheers,

 Jonathan
_______________________________________________
Zope-Dev maillist  -  Zope-Dev@...
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )

[Jonathan Ballet-5]

Hello,

On Tue, Nov 3, 2009 at 11:21 AM, Jonathan Ballet
<jonathan.ballet@...> wrote:

> Hello Christian,
>
> On Tue, Nov 3, 2009 at 11:07 AM, Christian Theune <ct@...> wrote:
>> [...] btw: do you log the seed if
>> you generated one?)
>
> Yep, the seed is displayed at the end of tests, so that it can be
> reused to reproduce a failing test.
>
>    $ ./bin/test --shuffle -cvvv
>    Running tests at level 1
>    Running zope.testing.testrunner.layer.UnitTests tests:
>      Set up zope.testing.testrunner.layer.UnitTests in 0.000 seconds.
>      ...
>      Ran 257 tests with 0 failures and 0 errors in 21.908 seconds.
>    Tests were shuffled using seed number 321854366611

Any thoughts on this feature? Did someone try it?
Is there any possibilities to merge this into trunk and to release a
new version with this?

Cheers,

 Jonathan
_______________________________________________
Zope-Dev maillist  -  Zope-Dev@...
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )

[Christian Theune-2]

Hi there,

On 11/23/2009 10:27 AM, Jonathan Ballet wrote:

> Hello,
>
> On Tue, Nov 3, 2009 at 11:21 AM, Jonathan Ballet
> <jonathan.ballet@...>  wrote:
>> Hello Christian,
>>
>> On Tue, Nov 3, 2009 at 11:07 AM, Christian Theune<ct@...>  wrote:
>>> [...] btw: do you log the seed if
>>> you generated one?)
>>
>> Yep, the seed is displayed at the end of tests, so that it can be
>> reused to reproduce a failing test.
>>
>>     $ ./bin/test --shuffle -cvvv
>>     Running tests at level 1
>>     Running zope.testing.testrunner.layer.UnitTests tests:
>>       Set up zope.testing.testrunner.layer.UnitTests in 0.000 seconds.
>>       ...
>>       Ran 257 tests with 0 failures and 0 errors in 21.908 seconds.
>>     Tests were shuffled using seed number 321854366611
>
> Any thoughts on this feature? Did someone try it?
> Is there any possibilities to merge this into trunk and to release a
> new version with this?

It's still on my todo list and I'll get to it at some point ... maybe
even during this week. ;)

Christian

--
Christian Theune · ct@...
gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany
http://gocept.com · tel [phone] · fax [phone]
Zope and Plone consulting and development

_______________________________________________
Zope-Dev maillist  -  Zope-Dev@...
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )