How to test 3rd party framework changes to webservice behavior?

[nhajratw]

Here is the scenario:

        * We have some webservices that rely on a 3rd party framework (Apache  
CXF) to do the marshaling to/from JSON into Java classes.
        * The webservices are invoked from a Javascript-based UI (ajax/etc).
        * We have no automated through-the-UI tests.
        * We have a set of automated integration tests that run against the  
services on a deployed server via HTTP requests.
        * We have a set of isolated unit tests that run against the business  
logic

Over the weekend, CXF was upgraded from 2.2.3 to 2.2.4. This upgrade  
appears to have caused more strict checking of incoming JSON  
parameters. i.e. Previously, null was allowed to be passed in and was  
silently ignored, and now null throws a Marshaling Exception.

We can (and did) update our integration tests to take this into  
account, but the UI ended up failing since it didn't know how to deal  
with the Marshaling Exception.

We're trying to move more our testing into isolated, automated unit  
tests, rather than integration tests, but i'm not sure how we would be  
able to catch problems like this without through-the-UI testing.

Questions:

        * Do we need to put more effort into automating through-the-UI testing?
        * Do we need to write tests on the Marshaling behavior of the  
framework?
        * Can our goal be accomplished without resorting to more integration  
tests?

Thanks!

---
Nayan Hajratwala
http://agileshrugged.com
http://twitter.com/nhajratw
734.658.6032

[Ashwini M]

Normally, upgrades are backward compatible, so there should be no issues.

However, in 1% of the cases when the interface or the functionality changes, you can do something:
1. stop automatic updates
2. do a proof of concept test on a new update when it arrives
3. if the test passes, update your server/client machines.

--- In agile-testing@..., Nayan Hajratwala <nayan@...> wrote:

>
> Here is the scenario:
>
> * We have some webservices that rely on a 3rd party framework (Apache  
> CXF) to do the marshaling to/from JSON into Java classes.
> * The webservices are invoked from a Javascript-based UI (ajax/etc).
> * We have no automated through-the-UI tests.
> * We have a set of automated integration tests that run against the  
> services on a deployed server via HTTP requests.
> * We have a set of isolated unit tests that run against the business  
> logic
>
> Over the weekend, CXF was upgraded from 2.2.3 to 2.2.4. This upgrade  
> appears to have caused more strict checking of incoming JSON  
> parameters. i.e. Previously, null was allowed to be passed in and was  
> silently ignored, and now null throws a Marshaling Exception.
>
> We can (and did) update our integration tests to take this into  
> account, but the UI ended up failing since it didn't know how to deal  
> with the Marshaling Exception.
>
> We're trying to move more our testing into isolated, automated unit  
> tests, rather than integration tests, but i'm not sure how we would be  
> able to catch problems like this without through-the-UI testing.
>
> Questions:
>
> * Do we need to put more effort into automating through-the-UI testing?
> * Do we need to write tests on the Marshaling behavior of the  
> framework?
> * Can our goal be accomplished without resorting to more integration  
> tests?
>
> Thanks!
>
> ---
> Nayan Hajratwala
> http://agileshrugged.com
> http://twitter.com/nhajratw
> 734.658.6032
>

[Markus Gaertner]

Nayan,

personally I would like to point you out to Mike Cohn's Testing Pyramid:
- you need lots and lots of unit tests
- you need some acceptance tests
- you need some through-the-UI tests
plus you need Exploratory Tests.

Regarding third party dependencies I think that the code you're testing is not
very well encapsulating the 3rd party framework - this is an assumption from
me, could be any other reason, as well. Do you need the upgrades to version
2.2.4? If not, maybe it's wise to first of all refactor the program for easier
adaptation of the changes. Maybe it's a story from your product owner, maybe
you need the functionality from the 3rd party library (obviously, I don't know
Apache CXF).

In any case, you seemed to have found a problem with your current approach. I
would try to get to the root cause of the problem, identifiy it, stress it out
and learn from it. Seek for the best opportunity and implement it. In general
it seems to me that you identified a problem with the software due to a new
third-party library. Are you also having similar problems with test fixture
adaptations there? If not, go out, help your developers to adapt to the new
situation, maybe encapsulating these changing components for the next version
upgrade.

Hope that helps a bit.

Kind regards
Markus Gärtner


Nayan Hajratwala wrote:

> Here is the scenario:
>
>     * We have some webservices that rely on a 3rd party framework
> (Apache CXF) to do the marshaling to/from JSON into Java classes.
>     * The webservices are invoked from a Javascript-based UI (ajax/etc).
>     * We have no automated through-the-UI tests.
>     * We have a set of automated integration tests that run against the
> services on a deployed server via HTTP requests.
>     * We have a set of isolated unit tests that run against the business
> logic
>
> Over the weekend, CXF was upgraded from 2.2.3 to 2.2.4. This upgrade
> appears to have caused more strict checking of incoming JSON parameters.
> i.e. Previously, null was allowed to be passed in and was silently
> ignored, and now null throws a Marshaling Exception.
>
> We can (and did) update our integration tests to take this into account,
> but the UI ended up failing since it didn't know how to deal with the
> Marshaling Exception.
>
> We're trying to move more our testing into isolated, automated unit
> tests, rather than integration tests, but i'm not sure how we would be
> able to catch problems like this without through-the-UI testing.
>
> Questions:
>
>     * Do we need to put more effort into automating through-the-UI testing?
>     * Do we need to write tests on the Marshaling behavior of the
> framework?
>     * Can our goal be accomplished without resorting to more integration
> tests?
>
> Thanks!
>
> ---
> Nayan Hajratwala
> http://agileshrugged.com
> http://twitter.com/nhajratw
> 734.658.6032
>
>

[nhajratw]

On Oct 19, 2009, at 12:51 PM, Markus Gaertner wrote:

Thanks, Markus...

> personally I would like to point you out to Mike Cohn's Testing  
> Pyramid:
> - you need lots and lots of unit tests
> - you need some acceptance tests
> - you need some through-the-UI tests
> plus you need Exploratory Tests.

Yes, I am familiar with the pyramid. I'm trying to help out a team  
that previously had a couple unit tests, lots of failing integration  
tests, some efforts at starting automated UI tests, and lots of mostly  
un-run manual UI tests.

We're on a good path to "moving down the pyramid", i'm just not sure  
how to address this particular situation without significantly  
expanding the top of the pyramid.


> Regarding third party dependencies I think that the code you're  
> testing is not
> very well encapsulating the 3rd party framework - this is an  
> assumption from
> me, could be any other reason, as well.

This is correct. The framework intercepts HTTP Requests, does the  
unmarshalling and hands the Java objects to the business code. It does  
the reverse for responses.  The normal use-case for this framework  
does not include putting a wrapper around it, as it is supposed to  
*be* the wrapper...

> Do you need the upgrades to version
> 2.2.4? If not, maybe it's wise to first of all refactor the program  
> for easier
> adaptation of the changes. Maybe it's a story from your product  
> owner, maybe
> you need the functionality from the 3rd party library (obviously, I  
> don't know
> Apache CXF).

There was no specific reason for moving to 2.2.3 for this release,  
other than to stay current, but there are often bugfixes and new  
features that we are bringing in. We've also found it easier to keep  
up with minor versions incrementally than to try to upgrade a major  
version after a long wait.

> In any case, you seemed to have found a problem with your current  
> approach.

Yes :-)

> I would try to get to the root cause of the problem, identifiy it,  
> stress it out
> and learn from it. Seek for the best opportunity and implement it.

Yes, that is the plan...

> Are you also having similar problems with test fixture adaptations  
> there?

Changing the test fixtures to handle the new case was not difficult at  
all.

> Hope that helps a bit.

A little, but I'm hoping that someone might have some concrete  
strategies.

>
> Nayan Hajratwala wrote:
> > Here is the scenario:
> >
> > * We have some webservices that rely on a 3rd party framework
> > (Apache CXF) to do the marshaling to/from JSON into Java classes.
> > * The webservices are invoked from a Javascript-based UI (ajax/etc).
> > * We have no automated through-the-UI tests.
> > * We have a set of automated integration tests that run against the
> > services on a deployed server via HTTP requests.
> > * We have a set of isolated unit tests that run against the business
> > logic
> >
> > Over the weekend, CXF was upgraded from 2.2.3 to 2.2.4. This upgrade
> > appears to have caused more strict checking of incoming JSON  
> parameters.
> > i.e. Previously, null was allowed to be passed in and was silently
> > ignored, and now null throws a Marshaling Exception.
> >
> > We can (and did) update our integration tests to take this into  
> account,
> > but the UI ended up failing since it didn't know how to deal with  
> the
> > Marshaling Exception.
> >
> > We're trying to move more our testing into isolated, automated unit
> > tests, rather than integration tests, but i'm not sure how we  
> would be
> > able to catch problems like this without through-the-UI testing.
> >
> > Questions:
> >
> > * Do we need to put more effort into automating through-the-UI  
> testing?
> > * Do we need to write tests on the Marshaling behavior of the
> > framework?
> > * Can our goal be accomplished without resorting to more integration
> > tests?
> >
> > Thanks!
>

[Markus Gaertner]

Ideally what I would try to achieve is a separation from the interface as much
as possible, while leaving some of the through-the-UI tests in there. It seems
to be possible to circumvent the JSON frontend at least for unit tests and for
some of the FitNesse tests as well by providing the Java objects for the
business logic directly. However, this will leave that risk that you need to
test the application from end-to-end, still, i.e. using exploratory tests.
Pushing down the test strategy to some level more down in the application could
turn out to be good here, iff your developers also help you with this.

Kind regards
Markus Gärtner

Nayan Hajratwala wrote:

> On Oct 19, 2009, at 12:51 PM, Markus Gaertner wrote:
>
> Thanks, Markus...
>
>> personally I would like to point you out to Mike Cohn's Testing  
>> Pyramid:
>> - you need lots and lots of unit tests
>> - you need some acceptance tests
>> - you need some through-the-UI tests
>> plus you need Exploratory Tests.
>
> Yes, I am familiar with the pyramid. I'm trying to help out a team  
> that previously had a couple unit tests, lots of failing integration  
> tests, some efforts at starting automated UI tests, and lots of mostly  
> un-run manual UI tests.
>
> We're on a good path to "moving down the pyramid", i'm just not sure  
> how to address this particular situation without significantly  
> expanding the top of the pyramid.
>
>
>> Regarding third party dependencies I think that the code you're  
>> testing is not
>> very well encapsulating the 3rd party framework - this is an  
>> assumption from
>> me, could be any other reason, as well.
>
> This is correct. The framework intercepts HTTP Requests, does the  
> unmarshalling and hands the Java objects to the business code. It does  
> the reverse for responses.  The normal use-case for this framework  
> does not include putting a wrapper around it, as it is supposed to  
> *be* the wrapper...
>
>> Do you need the upgrades to version
>> 2.2.4? If not, maybe it's wise to first of all refactor the program  
>> for easier
>> adaptation of the changes. Maybe it's a story from your product  
>> owner, maybe
>> you need the functionality from the 3rd party library (obviously, I  
>> don't know
>> Apache CXF).
>
> There was no specific reason for moving to 2.2.3 for this release,  
> other than to stay current, but there are often bugfixes and new  
> features that we are bringing in. We've also found it easier to keep  
> up with minor versions incrementally than to try to upgrade a major  
> version after a long wait.
>
>> In any case, you seemed to have found a problem with your current  
>> approach.
>
> Yes :-)
>
>> I would try to get to the root cause of the problem, identifiy it,  
>> stress it out
>> and learn from it. Seek for the best opportunity and implement it.
>
> Yes, that is the plan...
>
>> Are you also having similar problems with test fixture adaptations  
>> there?
>
> Changing the test fixtures to handle the new case was not difficult at  
> all.
>
>> Hope that helps a bit.
>
> A little, but I'm hoping that someone might have some concrete  
> strategies.
>
>
>> Nayan Hajratwala wrote:
>>> Here is the scenario:
>>>
>>> * We have some webservices that rely on a 3rd party framework
>>> (Apache CXF) to do the marshaling to/from JSON into Java classes.
>>> * The webservices are invoked from a Javascript-based UI (ajax/etc).
>>> * We have no automated through-the-UI tests.
>>> * We have a set of automated integration tests that run against the
>>> services on a deployed server via HTTP requests.
>>> * We have a set of isolated unit tests that run against the business
>>> logic
>>>
>>> Over the weekend, CXF was upgraded from 2.2.3 to 2.2.4. This upgrade
>>> appears to have caused more strict checking of incoming JSON  
>> parameters.
>>> i.e. Previously, null was allowed to be passed in and was silently
>>> ignored, and now null throws a Marshaling Exception.
>>>
>>> We can (and did) update our integration tests to take this into  
>> account,
>>> but the UI ended up failing since it didn't know how to deal with  
>> the
>>> Marshaling Exception.
>>>
>>> We're trying to move more our testing into isolated, automated unit
>>> tests, rather than integration tests, but i'm not sure how we  
>> would be
>>> able to catch problems like this without through-the-UI testing.
>>>
>>> Questions:
>>>
>>> * Do we need to put more effort into automating through-the-UI  
>> testing?
>>> * Do we need to write tests on the Marshaling behavior of the
>>> framework?
>>> * Can our goal be accomplished without resorting to more integration
>>> tests?
>>>
>>> Thanks!
>
>

[nails762]

--- In agile-testing@..., Nayan Hajratwala <nayan@...> wrote:

>
> Here is the scenario:
>
> * We have some webservices that rely on a 3rd party framework (Apache  
> CXF) to do the marshaling to/from JSON into Java classes.
> * The webservices are invoked from a Javascript-based UI (ajax/etc).
> * We have no automated through-the-UI tests.
> * We have a set of automated integration tests that run against the  
> services on a deployed server via HTTP requests.
> * We have a set of isolated unit tests that run against the business  
> logic
>
> Over the weekend, CXF was upgraded from 2.2.3 to 2.2.4. This upgrade  
> appears to have caused more strict checking of incoming JSON  
> parameters. i.e. Previously, null was allowed to be passed in and was  
> silently ignored, and now null throws a Marshaling Exception.

I suppose you didn't have a Learning Test for CXF that showed that you depended on that behavior.

> We can (and did) update our integration tests to take this into  
> account, but the UI ended up failing since it didn't know how to deal  
> with the Marshaling Exception.

When you changed the behavior in one layer, you need to change the expected behavior (stubs, mocks, whatever) in that layer's clients.

> We're trying to move more our testing into isolated, automated unit  
> tests, rather than integration tests, but i'm not sure how we would be  
> able to catch problems like this without through-the-UI testing.

You changed the contract without reflecting that change in its clients. You need to remember to do that. There's no magic and nothing new there. I agree that through-the-UI testing would expose the error, but then, end-to-end testing would expose every error. :)

When you change the contract of an object, then you have to check its clients. When you rely on the contract of someone else's code, then you need to easily detect their changes to the contracts. You missed on both counts.

> * Do we need to put more effort into automating through-the-UI testing?

No, you don't need to do it. If you did, it might help, but I wouldn't consider ROI high enough to do it.

> * Do we need to write tests on the Marshaling behavior of the  
> framework?

I would do that. I would write Learning Tests for the parts of the Marshalling behavior on which I rely. These document the Marshalling behavior you can safely assume elsewhere in your tests.

> * Can our goal be accomplished without resorting to more integration  
> tests?

Yes, yes... oh god, yes!
----
J. B. (Joe) Rainsberger :: http://www.jbrains.ca :: http://blog.thecodewhisperer.com
Your guide to software craftsmanship
JUnit Recipes: Practical Methods for Programmer Testing
2005 Gordon Pask Award for contributions to Agile Software Practice

[nhajratw]

On Oct 19, 2009, at 1:26 PM, Markus Gaertner wrote:

> Ideally what I would try to achieve is a separation from the  
> interface as much
> as possible,

It appears to me that this is precisely what is *causing* my problem..  
The server side business logic has tests covering it. It is the  
marshaling of the HttpRequest containing JSON to the Java objects by  
the framework that is causing the issue.

Even if I write tests around the framework (i.e. validating what I  
expect to occur when I pass in null), how can I ensure that the UI  
will expect the same behavior when *it* passes in null?

> while leaving some of the through-the-UI tests in there. It seems
> to be possible to circumvent the JSON frontend at least for unit  
> tests and for
> some of the FitNesse tests as well by providing the Java objects for  
> the
> business logic directly. However, this will leave that risk that you  
> need to
> test the application from end-to-end, still, i.e. using exploratory  
> tests.
> Pushing down the test strategy to some level more down in the  
> application could
> turn out to be good here, iff your developers also help you with this.
>
> Kind regards
> Markus Gärtner
>

[nhajratw]

On Oct 19, 2009, at 1:32 PM, nails762 wrote:

> --- In agile-testing@..., Nayan Hajratwala <nayan@...>  
> wrote:
> >
> > Here is the scenario:
> >
> > * We have some webservices that rely on a 3rd party framework  
> (Apache
> > CXF) to do the marshaling to/from JSON into Java classes.
> > * The webservices are invoked from a Javascript-based UI (ajax/etc).
> > * We have no automated through-the-UI tests.
> > * We have a set of automated integration tests that run against the
> > services on a deployed server via HTTP requests.
> > * We have a set of isolated unit tests that run against the business
> > logic
> >
> > Over the weekend, CXF was upgraded from 2.2.3 to 2.2.4. This upgrade
> > appears to have caused more strict checking of incoming JSON
> > parameters. i.e. Previously, null was allowed to be passed in and  
> was
> > silently ignored, and now null throws a Marshaling Exception.
>
> I suppose you didn't have a Learning Test for CXF that showed that  
> you depended on that behavior.

You suppose correctly.

> > We can (and did) update our integration tests to take this into
> > account, but the UI ended up failing since it didn't know how to  
> deal
> > with the Marshaling Exception.
>
> When you changed the behavior in one layer, you need to change the  
> expected behavior (stubs, mocks, whatever) in that layer's clients.

Yes, that is correct.

> > We're trying to move more our testing into isolated, automated unit
> > tests, rather than integration tests, but i'm not sure how we  
> would be
> > able to catch problems like this without through-the-UI testing.
>
> You changed the contract without reflecting that change in its  
> clients. You need to remember to do that. There's no magic and  
> nothing new there. I agree that through-the-UI testing would expose  
> the error, but then, end-to-end testing would expose every error. :)
>
> When you change the contract of an object, then you have to check  
> its clients. When you rely on the contract of someone else's code,  
> then you need to easily detect their changes to the contracts. You  
> missed on both counts.

So this is the missing piece .. Being able to detect who the clients  
of the object (service) in question. Harder to do since these are  
dynamically generated JS calls made by the UI... but a missing piece  
indeed.

> > * Can our goal be accomplished without resorting to more integration
> > tests?
>
> Yes, yes... oh god, yes!

My bait worked :-)

Thanks JB!

[John Overbaugh]

Mayan you have a few options... What seems most applicable here is to  
include a series of page validations where you scan for errors (rather  
than rigorously validating functionality). As your web pages are  
really just streams of text, you could spider your site, then build a  
series of tests which grab each URL, request it from the server, and  
then scan the resulting stream with a regex looking for various  
strings indicating an error.

Seems relatively efficient--quicker than interactive UI automation,  
fast to develop.

Hope that helps!

John Overbaugh
Director of Quality Assurance
Medicity



On Oct 19, 2009, at 12:33 PM, "Ashwini M" <amahajan66@...> wrote:

> Normally, upgrades are backward compatible, so there should be no  
> issues.
>
> However, in 1% of the cases when the interface or the functionality  
> changes, you can do something:
> 1. stop automatic updates
> 2. do a proof of concept test on a new update when it arrives
> 3. if the test passes, update your server/client machines.
>
> --- In agile-testing@..., Nayan Hajratwala <nayan@...>  
> wrote:
> >
> > Here is the scenario:
> >
> > * We have some webservices that rely on a 3rd party framework  
> (Apache
> > CXF) to do the marshaling to/from JSON into Java classes.
> > * The webservices are invoked from a Javascript-based UI (ajax/etc).
> > * We have no automated through-the-UI tests.
> > * We have a set of automated integration tests that run against the
> > services on a deployed server via HTTP requests.
> > * We have a set of isolated unit tests that run against the business
> > logic
> >
> > Over the weekend, CXF was upgraded from 2.2.3 to 2.2.4. This upgrade
> > appears to have caused more strict checking of incoming JSON
> > parameters. i.e. Previously, null was allowed to be passed in and  
> was
> > silently ignored, and now null throws a Marshaling Exception.
> >
> > We can (and did) update our integration tests to take this into
> > account, but the UI ended up failing since it didn't know how to  
> deal
> > with the Marshaling Exception.
> >
> > We're trying to move more our testing into isolated, automated unit
> > tests, rather than integration tests, but i'm not sure how we  
> would be
> > able to catch problems like this without through-the-UI testing.
> >
> > Questions:
> >
> > * Do we need to put more effort into automating through-the-UI  
> testing?
> > * Do we need to write tests on the Marshaling behavior of the
> > framework?
> > * Can our goal be accomplished without resorting to more integration
> > tests?
> >
> > Thanks!
> >
> > ---
> > Nayan Hajratwala
> > http://agileshrugged.com
> > http://twitter.com/nhajratw
> > 734.658.6032
> >
>
>