Sharing/Reusing Expectations across tests.

[Chris Miles-5]

Hi,

When using the TestNG testing framework, I would like to use TestNG's
"dependsOnMethods" property to allow a test to rely on another test.

Say my class under test, and the method being tested does two sequential
events

public void process() {
    object1.doThis();
    object2.doThat();
}

So say I want to write two tests (correct me if there is a better design
here) to test that my class communicates with these two unrelated
collaborators.

Without test dependencies I would have 2 simple tests like this.

    @Test(groups = "unit")
    public void testDoThisCalled() {
        this.mockery.checking(new Expectations() {
            {
                oneOf(object1).doThis();
                ignoring(object2).doThat();
            }
        });

        this.myClass.process();
    }

    @Test(groups = "unit")
    public void testDoThatCalled() {
        this.mockery.checking(new Expectations() {
            {
                ignoring(object1).doThis();
                oneOf(object2).doThat();
            }
        });

        this.myClass.process();
    }

Each test tests the object in question, but I think it gets very messy
having to ignore all of the other objects. There is only two objects here,
but imagine ten or more, and having to ignore them for every test it gets
even more messy.

Now TestNG allows you to have one test as a dependent on another test, i.e
the setUp portion (@BeforeTest) is ran before the first test only so the
results of the first test can be used as part of the second test.

It would be good to be able to do something like

    @Test(groups = "unit")
    public void testDoThisCalled() {
        this.mockery.checking(new Expectations() {
            {
                oneOf(object1).doThis();
                ignoring(object2).doThat();
            }
        });

        this.myClass.process();
    }

    @Test(groups = "unit", dependsOnMethods = "testDoThisCalled")
    public void testDoThatCalled() {
        this.mockery.checking(new Expectations() {
            {
                oneOf(object2).doThat();
            }
        });

        this.myClass.process();
    }

Where the oneOf(object2).doThat() in the second test overwrites the
ignoring(object2).doThat() in the first test, and the
oneOf(object1).doThis() in test one is added to the expectations of the
second test removing the need to add it again. In this second example
TestNG runs test 1 and test 2 in the same sequence without reinitialising
things so I would like the exectations to be carried to the second test (I
hope that make sense)

Again, if there is a cleaner way of doing this I would be greatly
apreciated if I could be pointed in the right direction.

Best Regards,

Chris

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Nat Pryce]

Obviously we can't speak for JUnit.  If you want to discuss test
dependencies, you'd be better off doing that on the JUnit mailing
list.

2009/7/29  <chris@...>:
> Each test tests the object in question, but I think it gets very messy
> having to ignore all of the other objects.

That to me is a smell.  If I find that my tests look like this I start
asking questions about the design of the code under test:

* Why do I need to ignore all but one object per test?
* Why is the behaviour of the object under test so clearly split into
aspects that only involve a single one of its peers?
* How can I responsiblities between classes to make the objects more cohesive?

> There is only two objects here,
> but imagine ten or more, and having to ignore them for every test it gets
> even more messy.

That is an even bigger smell.  If I find set-up so arduous, I ask
myself questions like:

* Why does this object have so many collaborators?
* Is there a missing abstraction that can hide some of these
collaborators from the rest of the system?

--Nat

--
http://www.natpryce.com

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Nat Pryce]

Oops, I hit send to quickly!

But, given all that below, you can write helper methods or classes to
define expectations more easily.

For example:

private void ignoringAllExcept(final Object significant) {
    context.checking(new Expectations() {{
        for (Object o : allMockObjects) {
            if (o != significant) ignoring(o);
        }
    }});
}

Where allMockObjects is defined somewhere in the test, or you could
subclass Mockery to remember all the mock objects it has created.

--Nat

2009/7/29 Nat Pryce <nat.pryce@...>:

> Obviously we can't speak for JUnit.  If you want to discuss test
> dependencies, you'd be better off doing that on the JUnit mailing
> list.
>
> 2009/7/29  <chris@...>:
>> Each test tests the object in question, but I think it gets very messy
>> having to ignore all of the other objects.
>
> That to me is a smell.  If I find that my tests look like this I start
> asking questions about the design of the code under test:
>
> * Why do I need to ignore all but one object per test?
> * Why is the behaviour of the object under test so clearly split into
> aspects that only involve a single one of its peers?
> * How can I responsiblities between classes to make the objects more cohesive?
>
>> There is only two objects here,
>> but imagine ten or more, and having to ignore them for every test it gets
>> even more messy.
>
> That is an even bigger smell.  If I find set-up so arduous, I ask
> myself questions like:
>
> * Why does this object have so many collaborators?
> * Is there a missing abstraction that can hide some of these
> collaborators from the rest of the system?
>
> --Nat
>
> --
> http://www.natpryce.com
>

--
http://www.natpryce.com

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Chris Miles-5]

Nat,

Thanks. Basically the problem stems from the fact that the application I
am developing has to fit in with an existing API, in which my core class
will only be invoked with a single process() method. This process method
needs to do a bunch of stuff such as reading in a file and processing the
contents of it, and then notifying different listener roles of certain
things.

So I only have one public method in which I want to assert that a bunch of
different things are happning inside it. I cant see a way of refactoring
it in a way that for each step of important behaviour in the process()
method can be asserted and tested without having to repeat/include all the
assertions before and after.

Hope that makes sense.

Chris

> Oops, I hit send to quickly!
>
> But, given all that below, you can write helper methods or classes to
> define expectations more easily.
>
> For example:
>
> private void ignoringAllExcept(final Object significant) {
>     context.checking(new Expectations() {{
>         for (Object o : allMockObjects) {
>             if (o != significant) ignoring(o);
>         }
>     }});
> }
>
> Where allMockObjects is defined somewhere in the test, or you could
> subclass Mockery to remember all the mock objects it has created.
>
> --Nat
>
> 2009/7/29 Nat Pryce <nat.pryce@...>:
>> Obviously we can't speak for JUnit.  If you want to discuss test
>> dependencies, you'd be better off doing that on the JUnit mailing
>> list.
>>
>> 2009/7/29  <chris@...>:
>>> Each test tests the object in question, but I think it gets very messy
>>> having to ignore all of the other objects.
>>
>> That to me is a smell.  If I find that my tests look like this I start
>> asking questions about the design of the code under test:
>>
>> * Why do I need to ignore all but one object per test?
>> * Why is the behaviour of the object under test so clearly split into
>> aspects that only involve a single one of its peers?
>> * How can I responsiblities between classes to make the objects more
>> cohesive?
>>
>>> There is only two objects here,
>>> but imagine ten or more, and having to ignore them for every test it
>>> gets
>>> even more messy.
>>
>> That is an even bigger smell.  If I find set-up so arduous, I ask
>> myself questions like:
>>
>> * Why does this object have so many collaborators?
>> * Is there a missing abstraction that can hide some of these
>> collaborators from the rest of the system?
>>
>> --Nat
>>
>> --
>> http://www.natpryce.com
>>
>
>
>
> --
> http://www.natpryce.com
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>     http://xircles.codehaus.org/manage_email
>
>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Nat Pryce]

2009/7/29  <chris@...>:
> I only have one public method in which I want to assert that a bunch of
> different things are happning inside it. I cant see a way of refactoring
> it in a way that for each step of important behaviour in the process()
> method can be asserted and tested without having to repeat/include all the
> assertions before and after.

It sounds like you're focusing the test on the wrong thing. I would
not write a test for each "step" in the process method.  Rather I'd
focus it on each feature of the object as far as the client is
concerned, running (and testing) the entire process method from entry
to exit in each test.

You might have different tests for:

* possible paths through the conditionals in the process method.
* error cases: what happens when there is invalid input or dependencies fail
* how the process method notifies listener A
* how the process method notifies listener B

For the listener tests, I would only attach the relevant listener at
the start of that test.  That way, all other tests don't need to
explicitly ignore it.  For example:

@Test
public void notifiesSheepListenerWhenSheepProcessed() {
    final SheepListener sheepListener = context.mock(SheepListener.class);
    processor.addSheepListener(sheepListener);

    context.checking(new Expectations() {{
        oneOf(sheepListener).sheepProcessed();

        ... and expectations to make it process a sheep ...
    }});


    processor.process(); // maybe with arguments to make it process a sheep
}

--Nat
--
http://www.natpryce.com

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Nat Pryce]

The other thing I'd do is split the class into smaller component
classes and exhaustively test those, and then have a coarser grained
test around the processor itself that just checks that it composes its
components correctly.

--Nat

2009/7/29 Nat Pryce <nat.pryce@...>:

> 2009/7/29  <chris@...>:
>> I only have one public method in which I want to assert that a bunch of
>> different things are happning inside it. I cant see a way of refactoring
>> it in a way that for each step of important behaviour in the process()
>> method can be asserted and tested without having to repeat/include all the
>> assertions before and after.
>
> It sounds like you're focusing the test on the wrong thing. I would
> not write a test for each "step" in the process method.  Rather I'd
> focus it on each feature of the object as far as the client is
> concerned, running (and testing) the entire process method from entry
> to exit in each test.
>
> You might have different tests for:
>
> * possible paths through the conditionals in the process method.
> * error cases: what happens when there is invalid input or dependencies fail
> * how the process method notifies listener A
> * how the process method notifies listener B
>
> For the listener tests, I would only attach the relevant listener at
> the start of that test.  That way, all other tests don't need to
> explicitly ignore it.  For example:
>
> @Test
> public void notifiesSheepListenerWhenSheepProcessed() {
>    final SheepListener sheepListener = context.mock(SheepListener.class);
>    processor.addSheepListener(sheepListener);
>
>    context.checking(new Expectations() {{
>        oneOf(sheepListener).sheepProcessed();
>
>        ... and expectations to make it process a sheep ...
>    }});
>
>
>    processor.process(); // maybe with arguments to make it process a sheep
> }
>
> --Nat
> --
> http://www.natpryce.com
>

--
http://www.natpryce.com

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Chris Miles-5]

Nat,

thanks for the responses.

> It sounds like you're focusing the test on the wrong thing. I would
> not write a test for each "step" in the process method.  Rather I'd
> focus it on each feature of the object as far as the client is
> concerned, running (and testing) the entire process method from entry
> to exit in each test.

When each step is a logical function then this is where the problem is.

My current code looks like this roughly:



    public boolean process(Set<? extends TypeElement> annotations,
                           RoundEnvironment roundEnv) {
        // For each annotation type to be processed.
        for (TypeElement annotation : annotations) {
            // For each annotated element.
            for (Element element :
roundEnv.getElementsAnnotatedWith(annotation)) {
                // If the annotated element is a class.
                if (element.getKind().isClass()) {
                    // Process class element.
                    processClassElement(element);
                }
            }
        }

        // Always a true response
        return true;
    }

    /**
     * Process a class element.
     *
     * @param element Class element.
     */
    private void processClassElement(Element element) {
        processSource(element);
        // If there are class processors.
        if (this.classProcessors != null) {
            // with each class processor.
            for (ElementProcessor elementProcessor : this.classProcessors) {
                // ###### JUST ADDED THIS
                elementProcessor.setProcessingEnvironment(this.processingEnv);
                // Inject the source code reader.
                elementProcessor.setSourceCodeReader(this.sourceCodeReader);
                // Process this element.
                elementProcessor.process(element);
            }
        }
        else {
            // Raise compiler warning.
            this.processingEnv.getMessager().printMessage(
                    Diagnostic.Kind.WARNING, "No class processors found");
        }
    }

    /**
     * Processes the source code for a class element.
     *
     * @param element The class element.
     */
    private void processSource(Element element) {
        try {
            // Try and process any source file attached to the class element.
            Source source = this.sourceCodeReader.getSource(element);
            this.sourceCodeProcessor.setSource(source);
            this.sourceCodeProcessor.process();
        }
        catch (SourceReaderException e) {
            // Raise compiler warning.
            this.processingEnv.getMessager().printMessage(
                    Diagnostic.Kind.WARNING, "Problem reading source code");
        }
        catch (SourceProcessorException e) {
            // Raise compiler warning.
            this.processingEnv.getMessager().printMessage(
                    Diagnostic.Kind.WARNING, "Problem processing source
code");
        }
    }

Now I have tests which assert that each collaborator is invoked as
expected because I need to be
sure that this behaviour is always carried out.

And I have tests to make sure that exceptions thrown at any point are
handled correctly.

Every step above is carried out when the process() method is invoked. This
is part of the Java API
which I have no access to changing so need to accept. A list of mocks is
passed in, which is then
broken down into another list of mocks, which then a bunch of stuff need
to be done with.

So for an example, the test which gets to the end of the process looks like

   /**
     * Tests that the source code is processed.
     */
    @Test(groups = "unit")
    public void testSourceIsProcessed()
            throws SourceReaderException, SourceProcessorException {

        // Set up expectations.
        this.mockery.checking(new Expectations() {
            {
                // Get all annotated elements for this annotation.
                allowing(roundEnvironment).getElementsAnnotatedWith(annotation);
                will(returnValue(elements));

                // The first element is a class.
                allowing(element).getKind();
                will(returnValue(ElementKind.CLASS));

                // Ignore the class processor for this test.
                ignoring(classProcessor);

                // Expect the source code reader to return source object.
                allowing(sourceCodeReader).getSource(element);
                will(returnValue(source));

                // Ignore.
                ignoring(sourceCodeProcessor).setSource(source);

                // Process the source.
                oneOf(sourceCodeProcessor).process();
            }
        });

        // Process.
        this.coreClassElementProcessor.process(this.annotations,
                this.roundEnvironment);
    }


Where the process() on the source code processor is invoked is my
(current) final step.

So I have now came across a step in one of my collaborators where an
object in the parent
is required so I have added the bit above which says

                // ###### JUST ADDED THIS
                elementProcessor.setProcessingEnvironment(this.processingEnv);

My main question is, in order to add this one line, every single test now
fails with

"unexpected invocation:
elementProcessor.setProcessingEnvironment(<processingEnvironment>)"

So I then need to add the following to every single failing test.

// Ignore.
ignoring(elementProcessor).setProcessingEnvironmen(processingEnvironment);

So my tests are growing with all these "ignore" expectations, which is
making the tests look
complicated and unreadable. Regardless of what I focus my testing on, I am
still going to need
to ignore everything that happened before it.

And my class is very basic. I dont think this class can be made finer
grained without further
convoluting things.

I hope what I have written makes sense there, my head is spinning a little.

Chris

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email