100%% Method Testing and Generating Unit Tests

[nails762]

James P. Gunderson wrote:

> I think it is more difficult than that.
<snip />

Doubtless true, but when I find one solid difficulty, I stop searching
for more. :)

> Regardless of how well one
> program can reason about the behavior of another, it requires that the
> program reason about the intent of the original programmer. The
> automated tests _may_ be able to assess that the code is doing what it
> it written to do, but how can it determine that it wasn't written to do
> the right thing?

We can agree that no program can ever automatically assess the
programmer's intent, but it is possible to enumerate efficiently all
known (input, output) pairs for sufficiently simple methods, depending
on the size of the domain and range of the method. The problem is that
where I would really want automatic enumeration of all tests, it would
either be impossible (consider the case of depending on an interface) or
untimely (consider the case of billions of possible inputs).
--
J. B. (Joe) Rainsberger :: http://www.jbrains.ca
Your guide to software craftsmanship
JUnit Recipes: Practical Methods for Programmer Testing
2005 Gordon Pask Award for contribution Agile Software Practice

[nails762]

Kevin Lawrence wrote:

> As to JB's observation about whether it's possible to generate such
> tests for non-trivial programs - it's still early days for JUnit
> Factory but I think you'd be surprised by what's already possible.

I don't mean to imply that JUnitFactory's characterization tests aren't
important. They are crucial to help understand how software works,
although I wonder how many learning opportunities I would lose by having
most of those tests discovered for me over discovering them myself.
Writing characterization tests, themselves a specific kind of learning
test, is partly about the test and partly about the learning by writing it.

I think JUnitFactory is a wonderful idea, and I'd love to see it in use
on any Java project. (I was going to write "legacy Java project", but is
there any other kind?)

Take care.
--
J. B. (Joe) Rainsberger :: http://www.jbrains.ca
Your guide to software craftsmanship
JUnit Recipes: Practical Methods for Programmer Testing
2005 Gordon Pask Award for contribution Agile Software Practice

[nails762]

Ole Ersoy wrote:

> "What does 100% method unit testing mean?"

I think the answer is simple, but enforcing the answer is impossible.

     A method is 100% tested if (perhaps and only if) for any change
     I make to that method that causes its behavior to deviate from
     what I wanted, a test fails.

The problem with the definition is "from what I want". I'm willing to
bet that no program will be capable of this in my lifetime.
--
J. B. (Joe) Rainsberger :: http://www.jbrains.ca
Your guide to software craftsmanship
JUnit Recipes: Practical Methods for Programmer Testing
2005 Gordon Pask Award for contribution Agile Software Practice

[nails762]

Ole Ersoy wrote:

> Assuming the code is correct, it's possible to
> generate the "100% enumerated" set of tests.

Suppose your method depends on an instance of an interface. How do you
intend to enumerate tests that depend on the behavior of that interface?
In Java, I'm aware of no way to write code to describe the intended
behavior of an interface method.
--
J. B. (Joe) Rainsberger :: http://www.jbrains.ca
Your guide to software craftsmanship
JUnit Recipes: Practical Methods for Programmer Testing
2005 Gordon Pask Award for contribution Agile Software Practice

[nails762]

Ole Ersoy wrote:

> J.B.
>
> If you look at:
>
> http://pmd.sourceforge.net/ <http://pmd.sourceforge.net/>
>
> (Joakim - U da man)
>
> You'll see that there's gazillions of things that a
> program can reason about another program.

PMD rules are about structure, not behavior. When we write tests, we
mostly test behavior, not structure. One counterexample is using jdepend
to cause a test to fail when we introduce an unwanted cyclic dependency,
but those tests remain relatively rare.

> So that can be easily tested 100%. I showed how in
> some of the earlier posts with simple examples. I
> could make the examples more elaborate, but they would
> just be extensions of the simple example. The
> approach is what is important.

See my other message asking about methods that depend on instances of
interfaces. I'm interested to know how you'd extend your approach to
cover that case, because I genuinely do not see how to do it.

Take care.
--
J. B. (Joe) Rainsberger :: http://www.jbrains.ca
Your guide to software craftsmanship
JUnit Recipes: Practical Methods for Programmer Testing
2005 Gordon Pask Award for contribution Agile Software Practice

[Cédric Beust ♔]

On 1/11/07, James P. Gunderson <jgunders@...> wrote:

> Well, you are probably right, if a single method had 5000 test cases,
> it might be a little overly complex.

I don't think a tool should ever generate 5000 test cases:  it should
generate one data-driven test case and a data file that contains 5000
possible values for that test.

--
Cédric
http://testng.org


[Non-text portions of this message have been removed]

[Lasse Koskela]

On 1/11/07, J. B. Rainsberger <jbrains762@...> wrote:
>  I think JUnitFactory is a wonderful idea, and I'd love to see it in use
>  on any Java project. (I was going to write "legacy Java project", but is
>  there any other kind?)

A green field Java project just about to start?

Lasse

[Ole]

Hehe,

Got some good points there James.

For me a good design is easy to validate anyways :-)

Yeah I love the idea of the guidance part.

I think someone should be able to write down a
specification for something / a solution / and have
the code be automatically generated.

Eclipse EMF is a great example.

A model can be created in XML Schema, Using Java
Interfaces, XMI, etc. and the corresponding code for
it generated, along with an application layer and
presentation layer (Editor), + a testing stub project.

Then OCL can be used to specify method bodys, thus a
complete application can be generated in java without
any java coding.  Now the user interface is staticly
built, but I'm sure someone is doing something about
that.

Cheers,
- Ole





--- "James P. Gunderson" <jgunders@...>
wrote:

> Ole:
>
> Ole Ersoy wrote:
>  >
>  >
>  > I would think though that if the tool
>  > produced 5000 test cases to be manually
>  > checked, this is a great indication
>  > that the method should be refactored
>  > to reduce the number of possible combinations
>  > of input and output values that are of interest.
>
> Well, you are probably right, if a single method
> had 5000 test cases,
> it might be a little overly complex.  But, in the
> spirit of our mythical
> programmer who writes her code and then presses the
> 'Generate Tests'
> button; it would not be unlikely for the entire
> package to have 5000
> tests generated. This is great - we want lots of
> tests - provided we
> don't have to go through all the passing tests to
> see which are
> correctly testing invalid code. Clearly she is going
> to do incremental
> build and test, but all those tests have to be
> looked at eventually.
> That is why I proposed the guidance to tell the test
> generator 'I don't
> care what I actually wrote, this is what the code is
> supposed to do.'
> With the inclusion of that kind of guidance, the
> 100% testing with zero
> (programmer) effort may be possible.
>
>  > This fits well with the notion that testing is
> about
>  > good design.
>  >
>  > Good design is easy to validate by the developer.
>
> On an academic note, are you sure that "good design
> is easy to
> validate"?  I've certainly been in some animated
> discussions in which it
> was hard to reach agreement :-)
>
> Thanks,
> Jim
> --
> James P. Gunderson, Ph.D.                Cognitive
> Systems Architect
> jgunders@...                   Gamma Two,
> Inc.
> www.gamma-two.com             303.778.7400
>

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> "A computer is like an Old Testament god, with a lot
> of rules and no
> mercy." -- Joseph Campbell
>



 
____________________________________________________________________________________
Want to start your own business?
Learn how on Yahoo! Small Business.
http://smallbusiness.yahoo.com/r-index

[James P Gunderson]

J. B. Rainsberger wrote:

 > Doubtless true, but when I find one solid difficulty, I stop searching
 > for more. :)
 >
Well, by looking a the whole set of discovered difficulties, I have,
occasionally, found a solution that addressed several of them in a
manner that was better than the proposed solutions to the individual
difficulties; but, in general, I agree with you. I feel obligated to
throw in a bunch of computer science buzzwords (greedy algorithms,
NP-complete, etc.) This is, undoubtedly, a side effect of poor early
training :)

 >
 > We can agree that no program can ever automatically assess the
 > programmer's intent, but it is possible to enumerate efficiently all
 > known (input, output) pairs for sufficiently simple methods, depending
 > on the size of the domain and range of the method. The problem is that
 > where I would really want automatic enumeration of all tests, it would
 > either be impossible (consider the case of depending on an interface)
 > or
 > untimely (consider the case of billions of possible inputs).

I'm not sure I proposed exhaustive testing of every possible input
combination, all I wanted to do was let the characterization tests make
sure the software was doing something correctly, and add a set of
guidance tests to allow the test generator to also test that it is doing
the correct thing.  This may still be a large set, but it is the same
problem faced by humans who develop a test set: "When do you have enough
tests?"  (Ooops, time for one more compsci buzzword: halting problem).
So, on the question of what is 100%, it depends on scope to a degree.
When do you stop testing your code, and begin testing the compiler, the
runtime machine, the library, etc.?

Perhaps a more reasonable goal would be an automated test generator that
would produce a test set that met or exceeded the tests produced by a
reasonably good human.  Essentially a 'best parctices' goal.

Jim
--
James P. Gunderson, Ph.D.                Cognitive Systems Architect
jgunders@...                   Gamma Two, Inc.
www.gamma-two.com             303.778.7400
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
I am a great believer in luck,
and I find the harder I work the more I have of it.
-- Thomas Jefferson

[Ole]

Amen to that!

--- Cédric Beust ♔  <cbeust@...> wrote:

> On 1/11/07, James P. Gunderson
> <jgunders@...> wrote:
>
> > Well, you are probably right, if a single method
> had 5000 test cases,
> > it might be a little overly complex.
>
> I don't think a tool should ever generate 5000 test
> cases:  it should
> generate one data-driven test case and a data file
> that contains 5000
> possible values for that test.
>
> --
> Cédric
> http://testng.org
>
>
> [Non-text portions of this message have been
> removed]
>
>

 
____________________________________________________________________________________
Do you Yahoo!?
Everyone is raving about the all-new Yahoo! Mail beta.
http://new.mail.yahoo.com

[James P Gunderson]

Cedric:
        I guess I look at 5000 tests as the static representation of what the
data-driven test generator dynamically produces. In either case, we need
5000 inputs, and 5000 expected values. In the situation under discussion
(where a test generator was testing the extant code, which might not
represent the desired behavior) there would still be 5000 results that
needed to be looked at by a human. Way too much work regardless of how
it is produced :-)

Jim

Cédric Beust ? wrote:

>
>
> On 1/11/07, James P. Gunderson <jgunders@...
> <mailto:jgunders%40gamma-two.com>> wrote:
>
>  > Well, you are probably right, if a single method had 5000 test cases,
>  > it might be a little overly complex.
>
> I don't think a tool should ever generate 5000 test cases: it should
> generate one data-driven test case and a data file that contains 5000
> possible values for that test.
>
> --
> Cédric
> http://testng.org <http://testng.org>
>
> [Non-text portions of this message have been removed]
>
>
>
> __________ NOD32 1972 (20070111) Information __________
>
> This message was checked by NOD32 antivirus system.
> http://www.eset.com

--
James P. Gunderson, Ph.D.                Cognitive Systems Architect
jgunders@...                   Gamma Two, Inc.
www.gamma-two.com             303.778.7400
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
I am a great believer in luck,
and I find the harder I work the more I have of it.
-- Thomas Jefferson

[Ole]

Hey JB,

I could comment, but I would have to do it with
examples and I think we are all starting to think the
same way about this.

I jumped back to this message after looking at another
one, so now I'm going to jump back to that one :-)

Cheers,
- Ole


--- "J. B. Rainsberger" <jbrains762@...> wrote:

> Ole Ersoy wrote:
>
> > "What does 100% method unit testing mean?"
>
> I think the answer is simple, but enforcing the
> answer is impossible.
>
>      A method is 100% tested if (perhaps and only
> if) for any change
>      I make to that method that causes its behavior
> to deviate from
>      what I wanted, a test fails.
>
> The problem with the definition is "from what I
> want". I'm willing to
> bet that no program will be capable of this in my
> lifetime.
> --
> J. B. (Joe) Rainsberger :: http://www.jbrains.ca
> Your guide to software craftsmanship
> JUnit Recipes: Practical Methods for Programmer
> Testing
> 2005 Gordon Pask Award for contribution Agile
> Software Practice
>

 
____________________________________________________________________________________
Want to start your own business?
Learn how on Yahoo! Small Business.
http://smallbusiness.yahoo.com/r-index

[Ole]

JB,

I assume you mean that there's an interface
that describes the return value and parameters
for the method?

Then there's a class that implements the interface.

I think your point is...how do I know what the method
body is, if I'm only given the interface?

True - I can't test much then.

I'm relying on the method body and using it as the
specification of what the programmer intended for the
method to do.  If what the programmers code produces
is different from what their intention was then the
tests are wrong as well.

So the dataset produced for the tests will be wrong.

I think that's what you are saying, and it's a good
point.

Cheers,
- Ole


--- "J. B. Rainsberger" <jbrains762@...> wrote:

> Ole Ersoy wrote:
>
> > Assuming the code is correct, it's possible to
> > generate the "100% enumerated" set of tests.
>
> Suppose your method depends on an instance of an
> interface. How do you
> intend to enumerate tests that depend on the
> behavior of that interface?
> In Java, I'm aware of no way to write code to
> describe the intended
> behavior of an interface method.
> --
> J. B. (Joe) Rainsberger :: http://www.jbrains.ca
> Your guide to software craftsmanship
> JUnit Recipes: Practical Methods for Programmer
> Testing
> 2005 Gordon Pask Award for contribution Agile
> Software Practice
>

 
____________________________________________________________________________________
Do you Yahoo!?
Everyone is raving about the all-new Yahoo! Mail beta.
http://new.mail.yahoo.com

[nails762]

Ole Ersoy wrote:

> I assume you mean that there's an interface
> that describes the return value and parameters
> for the method?
>
> Then there's a class that implements the interface.

Indeed, there are perhaps 5 or 50 classes that implement that interface.

> I think your point is...how do I know what the method
> body is, if I'm only given the interface?
>
> True - I can't test much then.

Thank you. I wanted to make sure I wasn't missing something important.

> I'm relying on the method body and using it as the
> specification of what the programmer intended for the
> method to do. If what the programmers code produces
> is different from what their intention was then the
> tests are wrong as well.
>
> So the dataset produced for the tests will be wrong.
>
> I think that's what you are saying, and it's a good
> point.

It is one of the things I'm saying, but even before we worry about
whether a method is implemented correctly according to the wishes of the
programmer, we need to worry about how much testable behavior we can
even infer from a correct implementation that depends on abstract method
definitions.
--
J. B. (Joe) Rainsberger :: http://www.jbrains.ca
Your guide to software craftsmanship
JUnit Recipes: Practical Methods for Programmer Testing
2005 Gordon Pask Award for contribution Agile Software Practice

[nails762]

Lasse Koskela wrote:

> On 1/11/07, J. B. Rainsberger <jbrains762@...
> <mailto:jbrains762%40gmail.com>> wrote:
>  > I think JUnitFactory is a wonderful idea, and I'd love to see it in use
>  > on any Java project. (I was going to write "legacy Java project", but is
>  > there any other kind?)
>
> A green field Java project just about to start?

Do people still do that? Weird.
--
J. B. (Joe) Rainsberger :: http://www.jbrains.ca
Your guide to software craftsmanship
JUnit Recipes: Practical Methods for Programmer Testing
2005 Gordon Pask Award for contribution Agile Software Practice

[nails762]

James P. Gunderson wrote:

> I'm not sure I proposed exhaustive testing of every possible input
> combination,
<snip />

You probably did not, but the thread did.

> ...all I wanted to do was let the characterization tests make
> sure the software was doing something correctly, and add a set of
> guidance tests to allow the test generator to also test that it is doing
> the correct thing. This may still be a large set, but it is the same
> problem faced by humans who develop a test set: "When do you have enough
> tests?" (Ooops, time for one more compsci buzzword: halting problem).
> So, on the question of what is 100%, it depends on scope to a degree.
> When do you stop testing your code, and begin testing the compiler, the
> runtime machine, the library, etc.?
>
> Perhaps a more reasonable goal would be an automated test generator that
> would produce a test set that met or exceeded the tests produced by a
> reasonably good human. Essentially a 'best parctices' goal.

I think I'd be happy with something that produced any tests of real
value, rather than routine ones I wouldn't write anyway. I just haven't
seen a tool capable of that so far, and these days, my attention is
focused elsewhere, so I'm no longer looking for them.
--
J. B. (Joe) Rainsberger :: http://www.jbrains.ca
Your guide to software craftsmanship
JUnit Recipes: Practical Methods for Programmer Testing
2005 Gordon Pask Award for contribution Agile Software Practice

[nails762]

Cédric Beust ♔ wrote:

>
>
> On 1/11/07, James P. Gunderson <jgunders@...
> <mailto:jgunders%40gamma-two.com>> wrote:
>
>  > Well, you are probably right, if a single method had 5000 test cases,
>  > it might be a little overly complex.
>
> I don't think a tool should ever generate 5000 test cases: it should
> generate one data-driven test case and a data file that contains 5000
> possible values for that test.

That's 5000 tests by my count, either way you build it.
--
J. B. (Joe) Rainsberger :: http://www.jbrains.ca
Your guide to software craftsmanship
JUnit Recipes: Practical Methods for Programmer Testing
2005 Gordon Pask Award for contribution Agile Software Practice

[Ole]

> It is one of the things I'm saying, but even before
> we worry about
> whether a method is implemented correctly according
> to the wishes of the
> programmer, we need to worry about how much testable
> behavior we can
> even infer from a correct implementation that
> depends on abstract method
> definitions.


True.

Personally I prefer to keep a method as simple as
possible so that each method addresses a specific
concern within the process of mapping some inputs to
some outputs, while catching and dealing with possible
exceptions that can occur during mapping (From JDBC or
webservices for instance) and the possibility of
inputs that don't pass invarient constraint
definitions.

This way I can get a good idea of what goes in, and
how it should come out, and how to enumerate that.

This makes me feel comfortable with respect to what I
just wrote.

So if I were to apply this type of approach to legacy
code, I think I would first run something like the
branch analyzers in PMD (Thanks Joakim), to determine
which methods needs simplification, then simplify, and
then generate unit tests.

Then I'd have to go through and analyze whether the
code does what the concept that is being coded
defines.

I think this is where the template approach comes in
that Jimm is working on.  For me this type of approach
means having a domain specific language, like OCL,
that further constrains what is possible in Java,
giving the solution designer a more limited set of
tools for expressing an idea, while at the same time
giving the designer an exhaustive set of tools within
a specific solution domain, like Data Mining for
instance.

Cheers,
- Ole




--- "J. B. Rainsberger" <jbrains762@...> wrote:

> Ole Ersoy wrote:
>
> > I assume you mean that there's an interface
> > that describes the return value and parameters
> > for the method?
> >
> > Then there's a class that implements the
> interface.
>
> Indeed, there are perhaps 5 or 50 classes that
> implement that interface.
>
> > I think your point is...how do I know what the
> method
> > body is, if I'm only given the interface?
> >
> > True - I can't test much then.
>
> Thank you. I wanted to make sure I wasn't missing
> something important.
>
> > I'm relying on the method body and using it as the
> > specification of what the programmer intended for
> the
> > method to do. If what the programmers code
> produces
> > is different from what their intention was then
> the
> > tests are wrong as well.
> >
> > So the dataset produced for the tests will be
> wrong.
> >
> > I think that's what you are saying, and it's a
> good
> > point.
>

 
> --
> J. B. (Joe) Rainsberger :: http://www.jbrains.ca
> Your guide to software craftsmanship
> JUnit Recipes: Practical Methods for Programmer
> Testing
> 2005 Gordon Pask Award for contribution Agile
> Software Practice
>



 
____________________________________________________________________________________
Do you Yahoo!?
Everyone is raving about the all-new Yahoo! Mail beta.
http://new.mail.yahoo.com

[Cédric Beust ♔]

On 1/14/07, J. B. Rainsberger <jbrains762@...> wrote:

> >     Cédric Beust ♔ wrote:
> > I don't think a tool should ever generate 5000 test cases: it should
> > generate one data-driven test case and a data file that contains 5000
> > possible values for that test.

>     That's 5000 tests by my count, either way you build it.

Sure, but only one test method, so much easier to maintain.

--
Cedric
http://testng.org


[Non-text portions of this message have been removed]

[Ole]

I'm piggybacking on the old thread but changed the
title a little.

Eclipse EMF generates your model for you, if you give
it an XML Schema, Annotated Java Interfaces, etc.

Then once the Model is generated, you can get specific
objects from the resource that contains them like this
(Gets the dependency management element of a Maven
POM):

                                                DependencyManagement dependencyManagement =
                                                        (DependencyManagement)
parentResource.getEObject("//@project/@dependencyManagement");

However it's quite possible that the return object is
null, and NPEs will get thrown if the client code
using the generated model api does not catch them.

In this type of scenario we could generate tests per a
designer rule that all NPE's caused by object lookup
via fragments should be caught and an informative
message displayed.

Thus I would call this JSR / Framework context
testing.

It enumerates various possibilities within a framework
that can go wrong and allows the "Project Manager" for
instance to specifiy rules across the board for code
quality requirements.

In this case the domain could have the narrow
definition of "model object lookup via fragments".

Jimm, I'm going to lookup getting a project like this
started on EMF corner, that also works with some of
the other ideas discussed here.  Let me know if this
might fit with your project.  If anyone else is
interested please let me know as well.

Cheers,
- Ole

--- Kevin Lawrence <kev.lawrence@...> wrote:

> On 1/10/07, James P. Gunderson
> <jgunders@...> wrote:
> > Kevin:
> >         I think these are good ideas, although if
> I was deluged with 5000 test
> > cases that I then had to validate manually, I'd
> probably not use it.
>
> Our posts crossed in the ether but I hope it was
> clear that I agree
> with you. We, at Agitar, have put a lot of effort
> into the
> verification scenario but we have not neglected the
> verification
> scenario either. We are tackling it on a number of
> fronts.
>
> One approach is similar to the approach you
> suggested where you seed
> the test generator with some known-to-be-correct
> cases. The problem
> with this approach is that, in the limit, it becomes
> easier to write
> the test manually than to prod the machine into
> finding the right
> case.  If the machine can take these suggestions and
> riff on them to
> find 'cases like them' it becomes more interesting
> though.
>
> Another approach is one that David Saff describes in
> his "The Practice
> of Theories" (http://www.tiny.cc/HttFz) paper in
> which the human
> proposes hypotheses and the test generator finds
> examples that confirm
> the hypothesis or that prove it to be false.
>
> Yet another approach - my favorite - is to have the
> machine generate
> lots of examples of input-output sets and make it
> easy for a human
> reader to review them and spot inconsistencies.
> Finding smart ways of
> filtering down all the possible examples and smart
> ways of presenting
> them to the user are the two main challenges in this
> approach.
>
> Meanwhile, we think there is value in pure
> regression testing too.
>
> Kevin
>
> http://www.junitfactory.com
> You send us code. We send you tests. For free.
>

 
____________________________________________________________________________________
Expecting? Get great news right away with email
Auto-Check.
Try the Yahoo! Mail Beta.
http://advision.webevents.yahoo.com/mailbeta/newmail_tools.html 


 
____________________________________________________________________________________
Be a PS3 game guru.
Get your game face on with the latest PS3 news and previews at Yahoo! Games.
http://videogames.yahoo.com/platform?platform=120121