z3c.testsetup and layers

[Jan-Wijbrand Kolman-3]

Hi,

I'm trying to grasp testing layers in general and functional testing
layers in particular - all from a z3c.testsetup perspective. Since it is
not all that clear in my head still, let met try to write things down
for others to comment on.

I also have some questions and suggestions for z3c.testsetup in particular.


The summary of test layers:
===========================

* Test layers is a feature of zope.testing.

* Test layers can be used to group tests that have a shared test setup
and teardown (the setup and teardown happen once for all the tests in
grouped in that layer).

* Test layers are classes and thus test layers can inherit from each
other.

* The zope.testing.testrunner will make sure that setup functions of
more generic layers are called before that of the specific layers and
the other way around for the teardown functions.

* The setup and teardown functions are to be defined as classmethods(!)
on the test layers.

* See also:

http://apidoc.zope.org/++apidoc++/Code/zope/testing/testrunner-layers-api.txt/index.html


So far so good.


The summary of functional test layers:
======================================

* Functional test layers is a feature of zope.app.testing (notice the
.app in there!).

* Test layers can be used to group tests that have a shared test setup
and teardown in the context of a shared component registry that has been
initialized according to a *.zcml file that can be specified for
functional test layers (the setup and teardown and the component
registry initialization happen once for all the tests in grouped in that
layer).

* Functional test layers are classes, however, for some reason (probably
historical), they're still old style classes.

* Still, since they're classes functional test layers can inherit from
each other.

* The setup and teardown methods are just that, methods (not class
methods!).

* The testrunner __cannot__ on its own make sure that the setup methods
of the more generic layers are called before that to the specific
layers, nor the other way around for the teardown methods.

* You have to make sure all setups and teardown methods in the test
layer inheritance hierarchy are called in the correct order::

   class MyFunctionalLayer(zope.app.testing.functional.ZCMLLayer):

       def setUp(self):
           super(MyFunctionalLayer, self).setUp()
           # and now my stuff

       def tearDown(self):
           # first my stuff
           super(MyFunctionalLayer, self).tearDown()

* This difference in layer behaviour is annoyingly confusing.


So far so good (albeit a bit annoying).


The summary of using (functional)test layers in z3c.testsetup:
==============================================================

There're three "directives" that z3c.testsetup recognizes for setting up
test layers:

1) :layer: <dotted.name.of.layer.def>

2) :zcml-layer: <ZCML_filename>

3) :functional-zcml-layer: <ZCML_filename>

Note that these layer-ish directives __only__ work in docttests!


Ad 1):
------

This is the most "generically" useful directive. By specifying a dotted
name to the layer, the test is grouped in that layer.

It can point to either a "normal" layer class (i.e. of the "non
functional" kind). Alternatively it can point to a layer instance if it
is to be a functional layer. Compare:

layers.py:

   class MyLayer(object):
     pass


   class MyFunctionalLayer(ZCMLLayer):
     pass

   my_functional_layer = MyFunctionalLayer(
     '/path/to/testing.zcml', __name__, 'My Functional Layer')


test_non_functional.txt:

   :layer: layers.MyLayer


test_functional.txt:

   :layer: layers.my_functional_layer

Note that if the layer definition is an instance of ZCMLLayer (directly
or indirectly), the tests are suited up in a
zope.app.testing.functional.FunctioalDocFileSuite

Ad 2 and 3):
------------

The functionality of both :zcml-layer: and :functional-zcml-layer: can
be accomplished by using :layer:

However, in case you just want to use a common *.zcml file for a group
of tests and not need to provide a common setup and teardown you can use
these directives.

The directives' differences are subtle - and not yet completely clear to
me. In case of :functional-zmcl-layer:

* There're more convenience functions available in the test's globals.
(what extra functions are these?)

* Somehow the setUp and tearDown functions (what setUp and tearDown
functions?) are wrapped in something (in what and why?).

* Anything more?


So far so... well... alright. The layer-ish directives were a source of
confusion for me and maybe for others too. I have couple of suggestions:


Suggestions
===========

* The :zcml-layer: and :functional-zcml-layer: directives have a
slightly confusing name. They really only point to a zcml file, not to a
layer definition like :layer: does.

I can imagine that :zcml-file: and :functional-zcml-file: would've been
less confusing names.

* The difference between :functional-zcml-layer: and :zcml-layer: is
really not all that obvious (at least to me). I wonder if we cannot just
have just one, with the behaviour of :functional-zcml-layer:?

Alternatively, the z3c.testsetup documentation could provides some
directions for choosing one over the other.

* z3c.testsetup should support the :layer: directives in unittests too.

* The functional layers and "normal" layers should probably behave
similar. This is not something z3c.testsetup can fix. I'll post a
message about this on zope-dev.


I hope my summary is correct and the suggestions make sense. I hope it
is somewhat complete as well as useful for others. I'd really like to
get feedback though.


regards,
jw



_______________________________________________
Grok-dev mailing list
Grok-dev@...
https://mail.zope.org/mailman/listinfo/grok-dev

[Uli Fouquet]

Hi JW,

First of all, many thanks for the lots of work you've done!

Am Dienstag, den 20.10.2009, 13:21 +0200 schrieb Jan-Wijbrand Kolman:

> I'm trying to grasp testing layers in general and functional testing
> layers in particular - all from a z3c.testsetup perspective. Since it is
> not all that clear in my head still, let met try to write things down
> for others to comment on.
>
> I also have some questions and suggestions for z3c.testsetup in particular.
>
>
> The summary of test layers:
> ===========================
>
> * Test layers is a feature of zope.testing.
>
> * Test layers can be used to group tests that have a shared test setup
> and teardown (the setup and teardown happen once for all the tests in
> grouped in that layer).
>
> * Test layers are classes and thus test layers can inherit from each
> other.
>
> * The zope.testing.testrunner will make sure that setup functions of
> more generic layers are called before that of the specific layers and
> the other way around for the teardown functions.
>
> * The setup and teardown functions are to be defined as classmethods(!)
> on the test layers.

One might add, that one can add a setup function for each test, but
that's not layer related.

> * See also:
>
> http://apidoc.zope.org/++apidoc++/Code/zope/testing/testrunner-layers-api.txt/index.html
>
>
> So far so good.
>
>
> The summary of functional test layers:
> ======================================
>
> * Functional test layers is a feature of zope.app.testing (notice the
> .app in there!).
>
> * Test layers can be used to group tests that have a shared test setup
> and teardown in the context of a shared component registry that has been
> initialized according to a *.zcml file that can be specified for
> functional test layers (the setup and teardown and the component
> registry initialization happen once for all the tests in grouped in that
> layer).
>
> * Functional test layers are classes, however, for some reason (probably
> historical), they're still old style classes.
>
> * Still, since they're classes functional test layers can inherit from
> each other.
>
> * The setup and teardown methods are just that, methods (not class
> methods!).
>
> * The testrunner __cannot__ on its own make sure that the setup methods
> of the more generic layers are called before that to the specific
> layers, nor the other way around for the teardown methods.
>
> * You have to make sure all setups and teardown methods in the test
> layer inheritance hierarchy are called in the correct order::
>
>    class MyFunctionalLayer(zope.app.testing.functional.ZCMLLayer):
>
>        def setUp(self):
>            super(MyFunctionalLayer, self).setUp()
>            # and now my stuff
>
>        def tearDown(self):
>            # first my stuff
>            super(MyFunctionalLayer, self).tearDown()
>
> * This difference in layer behaviour is annoyingly confusing.
>
>
> So far so good (albeit a bit annoying).
>
>
> The summary of using (functional)test layers in z3c.testsetup:
> ==============================================================
>
> There're three "directives" that z3c.testsetup recognizes for setting up
> test layers:
>
> 1) :layer: <dotted.name.of.layer.def>
>
> 2) :zcml-layer: <ZCML_filename>
>
> 3) :functional-zcml-layer: <ZCML_filename>
>
> Note that these layer-ish directives __only__ work in docttests!
>
>
> Ad 1):
> ------
>
> This is the most "generically" useful directive. By specifying a dotted
> name to the layer, the test is grouped in that layer.
>
> It can point to either a "normal" layer class (i.e. of the "non
> functional" kind). Alternatively it can point to a layer instance if it
> is to be a functional layer. Compare:
>
> layers.py:
>
>    class MyLayer(object):
>      pass
>
>
>    class MyFunctionalLayer(ZCMLLayer):
>      pass
>
>    my_functional_layer = MyFunctionalLayer(
>      '/path/to/testing.zcml', __name__, 'My Functional Layer')
>
>
> test_non_functional.txt:
>
>    :layer: layers.MyLayer
>
>
> test_functional.txt:
>
>    :layer: layers.my_functional_layer
>
> Note that if the layer definition is an instance of ZCMLLayer (directly
> or indirectly), the tests are suited up in a
> zope.app.testing.functional.FunctioalDocFileSuite
>
> Ad 2 and 3):
> ------------
>
> The functionality of both :zcml-layer: and :functional-zcml-layer: can
> be accomplished by using :layer:
>
> However, in case you just want to use a common *.zcml file for a group
> of tests and not need to provide a common setup and teardown you can use
> these directives.
>
> The directives' differences are subtle - and not yet completely clear to
> me. In case of :functional-zmcl-layer:
>
> * There're more convenience functions available in the test's globals.
> (what extra functions are these?)

'sync', 'http', 'getRootFolder', the usual suspects from
zope.app.testung.functional.

> * Somehow the setUp and tearDown functions (what setUp and tearDown
> functions?) are wrapped in something (in what and why?).

This is not the case any more, at least for tests registered with
[functional]-zcml-layer. When using the old test setup markers you got
your setup function be called by a function that also called
zope.app.testing.functional.FunctionalTestSetup().setUp().

Another confusion that arrived with the switch to the new markers :-/

> * Anything more?
>
>
> So far so... well... alright. The layer-ish directives were a source of
> confusion for me and maybe for others too. I have couple of suggestions:
>
>
> Suggestions
> ===========
>
> * The :zcml-layer: and :functional-zcml-layer: directives have a
> slightly confusing name. They really only point to a zcml file, not to a
> layer definition like :layer: does.
>
> I can imagine that :zcml-file: and :functional-zcml-file: would've been
> less confusing names.

Right. The reason for the name was that when using a 'zcml-file' or
'functional-zcml-layer' directive, you're supposed not to define any
additional layers as this does in fact define a layer for you. But
overall, I agree, it is more confusing.

> * The difference between :functional-zcml-layer: and :zcml-layer: is
> really not all that obvious (at least to me). I wonder if we cannot just
> have just one, with the behaviour of :functional-zcml-layer:?

Yeah, +1 from my side. Let's name it 'zcml-file'.

> Alternatively, the z3c.testsetup documentation could provides some
> directions for choosing one over the other.
>
> * z3c.testsetup should support the :layer: directives in unittests too.

You mean plain (non-doctest) unittests? I'd like to fiddle around with
them as little as possible. It should be possible to define a layer for
them in the code itself setting a ``layer`` attribute on the testsuite
or single tests.

> * The functional layers and "normal" layers should probably behave
> similar. This is not something z3c.testsetup can fix. I'll post a
> message about this on zope-dev.

Yes, it would be great if these two kinds could be more compatible.

> I hope my summary is correct and the suggestions make sense. I hope it
> is somewhat complete as well as useful for others. I'd really like to
> get feedback though.

Thank you very much for all the suggestions! I appreciate them very
much!

I think it's time to get rid of all the old stuff (the old markers) in
z3c.testsetup which makes things more complicated and blocks
readability. Cleaning up should lead to better understandable code and
behaviour. Doing this I could also rename the strange functional-layer
directives and reduce them to one: ':zcml-file:'.

I am still undecided about the default value of layer-teardown. Should
that be True, False or should we just use the default of zope.testing?
As there were recently some issues with it (again) I tend to reuse the
default of zope.testing.

Furthermore I am thinking about some (optional) manuel
(http://packages.python.org/manuel/) support. But that would need some
more work, maybe a redesign to make use of third-party testsetup code.

Best regards,

--
Uli



_______________________________________________
Grok-dev mailing list
Grok-dev@...
https://mail.zope.org/mailman/listinfo/grok-dev

[Reinout van Rees]

On 2009-10-21, Uli Fouquet <uli@...> wrote:
>
> I am still undecided about the default value of layer-teardown. Should
> that be True, False or should we just use the default of zope.testing?
> As there were recently some issues with it (again) I tend to reuse the
> default of zope.testing.

I agree.  Back to the default it is.

I've already made the change in svn yesterday evening.  This (and a typo fix
by jkolman) is the only change since 0.5.

Actually, I went ahead and made a 0.5.1 release to get rid of this issue
seeing that we all agree :-)


Reinout

--
Reinout van Rees - reinout@... - http://reinout.vanrees.org
Software developer at http://www.thehealthagency.com
"Military engineers build missiles. Civil engineers build targets"

_______________________________________________
Grok-dev mailing list
Grok-dev@...
https://mail.zope.org/mailman/listinfo/grok-dev

[Jan-Wijbrand Kolman-3]

Jan-Wijbrand Kolman wrote:
snip

> * You have to make sure all setups and teardown methods in the test
> layer inheritance hierarchy are called in the correct order::
>
>    class MyFunctionalLayer(zope.app.testing.functional.ZCMLLayer):
>
>        def setUp(self):
>            super(MyFunctionalLayer, self).setUp()
>            # and now my stuff
>
>        def tearDown(self):
>            # first my stuff
>            super(MyFunctionalLayer, self).tearDown()

I made a mistake in this example. Since ZCMLLayer is an old-style class,
you cannot use "super". The example should read:

    class MyFunctionalLayer(zope.app.testing.functional.ZCMLLayer):

        def setUp(self):
            zope.app.testing.functional.ZCMLLayer.setUp(self)
            # and now my stuff

        def tearDown(self):
            # first my stuff
            zope.app.testing.functional.ZCMLLayer.tearDown(self)


regards,
jw


_______________________________________________
Grok-dev mailing list
Grok-dev@...
https://mail.zope.org/mailman/listinfo/grok-dev