100%% Method Testing and Generating Unit Tests

[Ole]

Sorry Jimm - I've been writing to a bunch of Bobs
today,
and my reflexes kicked in when I saw the bobs...in
your email address...


--- Ole Ersoy <ole_ersoy@...> wrote:

> Bob,
>
> What you are describing sounds like it has a lot in
> common with this:
>
>
http://www.eclipse.org/articles/Article-EMF-Codegen-with-OCL/article.html

>
> EMF generates code for your model.
>
> However you can model invarient constraints, derived
> value operations, etc. as well.
>
> However without OCL EMF will just generate method
> stubs for overriding.
>
> With OCL EMF generates the entire method.
>
> So the type of template you are describing could be
> used to generate the OCL string as well.  With that
> the Operation could be generated.
>
> Then test cases could be generated that complement
> and
> prove that the method body generated based on the
> OCL
> string is correct, giving the developer further
> confidence in the solution.
>
> Just an idea.
>
> Cheers,
> - Ole
>
>
> --- bohs01@... wrote:
>
> > Yeap it aint what I had in mind, in fact i dont
> > think i can come up  
> > with something like it LoL
> >
> > What i had in mind was just some sort of template
> > for a user to fill  
> > in the blanks sortof
> >
> > and a tutorial which asks questions whose answers
> > are used to fill in  
> > the  above mentioned blanks
> >
> > Thus, in a way i think it helps with TDD, since it
> > drives the design  
> > process but asking questions like what are the pre
> > and post  
> > conditions, class invariants etc, that the user
> > needs to answer in  
> > order to generate the test cases.
> >
> > It also automates the generation of the test
> cases,
> > "generating of the  
> > source code" that is.
> >
> > yeap actually i've been reading the posts, and am
> > not sure what PMD  
> > does exactly...
> >
> > Jimm
> >
> >
> > Quoting Ole Ersoy <ole_ersoy@...>:
> >
> > > Wow - Dud - You Da Man!!
> > >
> > > (And judging by the name - quite possibly
> > Norwegian, which explains why
> > > u so da Man :-)  )
> > >
> > > This actually looks a lot like
> > > what I've been trying to describe,
> > > with the possible exception that I don't think
> > > it generates tests like Jimm wants to do,
> > > but hey this is real ;-).  HooohHahhh!
> > >
> > > Jimm in order for this to work the way it says
> it
> > > works it has to parse source and store it in
> some
> > > model.  Then analyze it...looks like using rule
> > sets.
> > >
> > > I'm sure you could use the same model and rule
> > sets
> > > to generate tests (Just shooting from the hip.
> > I'd have to look at it a
> > > little more
> > > to see whether I could come up with a concrete
> > example. )
> > >
> > > http://pmd.sourceforge.net/
> > >
> > > Thanks for letting me us Joakim.  Can't wait to
> > > to hook it up to my maven projects.
> > >
> > > Cheers,
> > > - Ole
> > >
> > >
> > >
> > >
> > > Joakim Ohlrogge wrote:
> > >>
> > >> > As a matter of fact...it would be nice to
> have
> > >> > something that enumerates the branches and
> > generates a
> > >> > report on which methods has a number of
> > branches over
> > >> > a certain limit...for code refactoring and
> > maintenance
> > >> > purposes.
> > >>
> > >> Hi Ole,
> > >>
> > >> I have not read the whole thread yet so if this
> > has already been
> > >> suggested I'm sorry.
> > >> PMD actually reports methods with a cyclomatic
> > complexity that is
> > >> larger than 10. If I'm not mistaken that is
> what
> > you're looking for.
> > >>
> > >> I find this helpful to find targets to refactor
> > in legacy code. When
> > >> TDD:ing the PLP (my abbreviation for path of
> > least pain) keeps the
> > >> cyclomatic complexity down (Like Jeff Langr
> > suggested earlier in the
> > >> threat).
> > >>
> > >> Cheers!
> > >> /J
> > >>
> > >>
> > >
> > >
> >
> >
> >
>
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam
> protection around
> http://mail.yahoo.com 
>

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com 

[nails762]

Ole Ersoy wrote:

> Oh - Maybe when I said,
>
> "Within the context of what has already been
> written..."
>
> you thought I meant
>
> "Within the context of what has already been written
> in this email alone..." :-)
>
> If that's the case then I can definitely see how that
> would be tricky :-)

No; I thought you meant I should go back and re-read what you'd written.
If you'd referred to a single e-mail, that would have helped, but I'm
not eager to fish for an answer I can't quickly find.

<snip />

> Here's a more general scenario.
>
> There's a developer. She fires up eclipse,
> creates a new class, and codes a method.
>
> Just 1.
>
> Then she selects the Eclipse menu (Hypothetical)
>
> Test > test
>
> and
>
> a src/main/test
>
> directory within her project is automatically created
> with tests that cover her method exhaustively (100%).
>
> So when I say 0 time for the developer, I mean 0 time
> coding the tests.
>
> Now she reviews the tests. Could be that she finds
> that her method needs some tweaking, to further
> contrain it's behaviour. Then she regenerates the
> tests.
>
> Etc.
>
> Now I think the question becomes:
>
> "Well, how do we know the test is tested
> 100%/Exhaustively?"
>
> This is where the part about reading some of the other
> stuff I wrote earlier in previous posts comes in.
> I'll be glad to elaborate on those :-)

I understand this better. It's a shame that such tools simply cannot
exist beyond the very simplest methods. If it were so easy to do, it
would be done. I'm no complexity/computational theory expert, but from
what I understand, there is very little any program can reason about the
behavior of any other program, so the tool you describe above probably
can't be useful for any non-trivial system.

As for the question of whether the method is tested exhaustively, that
depends on how well the test-generating code is tested... and then it's
turtles all the way down: an infinite regression. It simply pushes the
problem to the next programmer.
--
J. B. (Joe) Rainsberger :: http://www.jbrains.ca
Your guide to software craftsmanship
JUnit Recipes: Practical Methods for Programmer Testing
2005 Gordon Pask Award for contribution Agile Software Practice

[James P Gunderson]

I think it is more difficult than that. Regardless of how well one
program can reason about the behavior of another, it requires that the
program reason about the intent of the original programmer.  The
automated tests _may_ be able to assess that the code is doing what it
it written to do, but how can it determine that it wasn't written to do
the right thing?

We use unit tests to make sure that the software is doing what it is
doing the correct thing, as well as making sure that it does what it
does correctly.

Jim

J. B. Rainsberger wrote:
<snip>

>  > Now I think the question becomes:
>  >
>  > "Well, how do we know the test is tested
>  > 100%/Exhaustively?"
>  >
>  > This is where the part about reading some of the other
>  > stuff I wrote earlier in previous posts comes in.
>  > I'll be glad to elaborate on those :-)
>
> I understand this better. It's a shame that such tools simply cannot
> exist beyond the very simplest methods. If it were so easy to do, it
> would be done. I'm no complexity/computational theory expert, but from
> what I understand, there is very little any program can reason about the
> behavior of any other program, so the tool you describe above probably
> can't be useful for any non-trivial system.
>
> As for the question of whether the method is tested exhaustively, that
> depends on how well the test-generating code is tested... and then it's
> turtles all the way down: an infinite regression. It simply pushes the
> problem to the next programmer.
> --
> J. B. (Joe) Rainsberger :: http://www.jbrains.ca <http://www.jbrains.ca>
> Your guide to software craftsmanship
> JUnit Recipes: Practical Methods for Programmer Testing
> 2005 Gordon Pask Award for contribution Agile Software Practice

--
James P. Gunderson, Ph.D.                Cognitive Systems Architect
jgunders@...                   Gamma Two, Inc.
www.gamma-two.com             303.778.7400
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Intelligence is the ability to figure out a procedure rather than
the ability to carry one out.
  Peter Kugel, 1979

[Ole]

J.B.

If you look at:

http://pmd.sourceforge.net/

(Joakim - U da man)

You'll see that there's gazillions of things that a
program can reason about another program.

However, when it comes to method testing
we are only interested in what the allowed
set of input values are (Parameters) and whether
the return value that the method returns
matches precisely what we expect it to return.

So that can be easily tested 100%.  I showed how in
some of the earlier posts with simple examples.  I
could make the examples more elaborate, but they would
just be extensions of the simple example.  The
approach is what is important.

It could be that you are thinking OK how does this
scale with Database testing, Web Applications, etc.
but those are all specific testing domains with
specific method testing requirements that would be
designed around the JSR's for those domains, just like
pmd provides rule sets for specific domains/JSRs (JSF,
etc.).

Cheers,
Ole





--- "J. B. Rainsberger" <jbrains762@...> wrote:

> Ole Ersoy wrote:
>
> > Oh - Maybe when I said,
> >
> > "Within the context of what has already been
> > written..."
> >
> > you thought I meant
> >
> > "Within the context of what has already been
> written
> > in this email alone..." :-)
> >
> > If that's the case then I can definitely see how
> that
> > would be tricky :-)
>
> No; I thought you meant I should go back and re-read
> what you'd written.
> If you'd referred to a single e-mail, that would
> have helped, but I'm
> not eager to fish for an answer I can't quickly
> find.
>
> <snip />
>
> > Here's a more general scenario.
> >
> > There's a developer. She fires up eclipse,
> > creates a new class, and codes a method.
> >
> > Just 1.
> >
> > Then she selects the Eclipse menu (Hypothetical)
> >
> > Test > test
> >
> > and
> >
> > a src/main/test
> >
> > directory within her project is automatically
> created
> > with tests that cover her method exhaustively
> (100%).
> >
> > So when I say 0 time for the developer, I mean 0
> time
> > coding the tests.
> >
> > Now she reviews the tests. Could be that she finds
> > that her method needs some tweaking, to further
> > contrain it's behaviour. Then she regenerates the
> > tests.
> >
> > Etc.
> >
> > Now I think the question becomes:
> >
> > "Well, how do we know the test is tested
> > 100%/Exhaustively?"
> >
> > This is where the part about reading some of the
> other
> > stuff I wrote earlier in previous posts comes in.
> > I'll be glad to elaborate on those :-)
>
> I understand this better. It's a shame that such
> tools simply cannot
> exist beyond the very simplest methods. If it were
> so easy to do, it
> would be done. I'm no complexity/computational
> theory expert, but from
> what I understand, there is very little any program
> can reason about the
> behavior of any other program, so the tool you
> describe above probably
> can't be useful for any non-trivial system.
>
> As for the question of whether the method is tested
> exhaustively, that
> depends on how well the test-generating code is
> tested... and then it's
> turtles all the way down: an infinite regression. It
> simply pushes the
> problem to the next programmer.
> --
> J. B. (Joe) Rainsberger :: http://www.jbrains.ca
> Your guide to software craftsmanship
> JUnit Recipes: Practical Methods for Programmer
> Testing
> 2005 Gordon Pask Award for contribution Agile
> Software Practice
>

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com 

[Kevin Lawrence]

On 1/9/07, James P. Gunderson <jgunders@...> wrote:
> I think it is more difficult than that. Regardless of how well one
> program can reason about the behavior of another, it requires that the
> program reason about the intent of the original programmer.  The
> automated tests _may_ be able to assess that the code is doing what it
> it written to do, but how can it determine that it wasn't written to do
> the right thing?
>

There are lots of reasons why one might want to have tests. Off the
top of my head,

- to drive the design
- to document the design
- to demonstrate correctness
- to provide a safety net when changing unfamiliar code

This latter case, which Michael Feathers talks a great deal about in
WELC, is probably one of the most common reasons that people want
automated tests.

Putting it another way,

  if I change this line of code, what other behavior will change will
change as a result

We ask ourselves this question every time we maintain unfamiliar code
in an unfamiliar system without tests. The most conscientious
developers write tests before changing that line of code. Most don't.

This is such a common activity in software development that a program
that could characterize the behavior of another program and report
when the behavior has changed would be useful wouldn't it?

As to JB's observation about whether it's possible to generate such
tests for non-trivial programs - it's still early days for JUnit
Factory but I think you'd be surprised by what's already possible.


Kevin

www.junitfactory.com
You send us code. We send you tests. For free.

[Greg Irvine]

Kevin, I think you've missed James' point, at least in the snippet of his
email you've quoted.

 

He's not at all saying anything negative about automated testing.

He's saying that generating tests from existing code has the risk that the
code doesn't do what it was intended but a test is generated to pass based
on the incorrect code.  So you would have 100% test passes, which is
incorrect against the actual expected results against the requirements.

 

Some argue that false impressions are worse than no impression at all.  I'm
yet to decide on my position.

 

Greg.

 

 

 

  _____  

From: junit@... [mailto:junit@...] On Behalf Of
Kevin Lawrence
Sent: Wednesday, 10 January 2007 11:20 AM
To: junit@...
Subject: Re: [junit] 100%% Method Testing and Generating Unit Tests

 

On 1/9/07, James P. Gunderson <jgunders@gamma-
<mailto:jgunders%40gamma-two.com> two.com> wrote:
> I think it is more difficult than that. Regardless of how well one
> program can reason about the behavior of another, it requires that the
> program reason about the intent of the original programmer. The
> automated tests _may_ be able to assess that the code is doing what it
> it written to do, but how can it determine that it wasn't written to do
> the right thing?
>

There are lots of reasons why one might want to have tests. Off the
top of my head,

- to drive the design
- to document the design
- to demonstrate correctness
- to provide a safety net when changing unfamiliar code

This latter case, which Michael Feathers talks a great deal about in
WELC, is probably one of the most common reasons that people want
automated tests.

Putting it another way,

if I change this line of code, what other behavior will change will
change as a result

We ask ourselves this question every time we maintain unfamiliar code
in an unfamiliar system without tests. The most conscientious
developers write tests before changing that line of code. Most don't.

This is such a common activity in software development that a program
that could characterize the behavior of another program and report
when the behavior has changed would be useful wouldn't it?

As to JB's observation about whether it's possible to generate such
tests for non-trivial programs - it's still early days for JUnit
Factory but I think you'd be surprised by what's already possible.

Kevin

www.junitfactory.com
You send us code. We send you tests. For free.

 



[Non-text portions of this message have been removed]

[James P Gunderson]

Kevin:
   I totally agree that there are a number of reasons for developing and running tests. I think that your list is a great initial classification. I guess my point was focused on the suggestion that the programmer writes some code, and without providing any additional information, presses the button that generates all the test cases. JB brought up the question of how much one program can 'reason' about the behavior of another program. I agree with your observation that the ability to characterize a programs behavior is frequently needed and useful. I also agree that one package can do a lot to analyse another.

All I was pointing out is that the analyzing program only has the source code of the target software to work from. Yes, a lot can be done to make sure that that target program has no errors in it, but it is harder (without extra information) to assess whether the target code is doing the right thing.  For example, suppose I write a function to calculate the distance between to points in space. The analysis can make sure that corner cases are covered, the code has no unnecessary complexity, that all the code is reachable, and so forth. And this is really good, useful, and provides value to me the programmer.  However, if I program the distance calculation incorrectly (perhaps I calculate the manhattan distance instead of the euclidian) how does the analyzing program 'know' that I coded the wrong function?  Without the addition of annotations, invariants, or the inclusion of test data, the analyzing program is hobbled. However, if the programmer is required to add the additional information, this violates the notion that the programmer has to do zero addtional work.

Now, I would be happy to have a support system that provides all the benefits you outlined, and I would be perfectly happy to have a system that would take a small amount of additional info and demonstrate (dare we say prove) that the code I wrote is both error free and does the job I want it to. I was looking at the gap between the intent of the programmer and the function of the target software, and whether it is possible to bridge that gap without the programmer documenting the intent.

Jim


+++++
Kevin Lawrence wrote:
There are lots of reasons why one might want to have tests. Off the
top of my head,

- to drive the design
- to document the design
- to demonstrate correctness
- to provide a safety net when changing unfamiliar code

This latter case, which Michael Feathers talks a great deal about in
WELC, is probably one of the most common reasons that people want
automated tests.

Putting it another way,

  if I change this line of code, what other behavior will change will
change as a result

We ask ourselves this question every time we maintain unfamiliar code
in an unfamiliar system without tests. The most conscientious
developers write tests before changing that line of code. Most don't.

This is such a common activity in software development that a program
that could characterize the behavior of another program and report
when the behavior has changed would be useful wouldn't it?

As to JB's observation about whether it's possible to generate such
tests for non-trivial programs - it's still early days for JUnit
Factory but I think you'd be surprised by what's already possible.


Kevin

www.junitfactory.com
You send us code. We send you tests. For free.





[Non-text portions of this message have been removed]

[Kevin Lawrence]

On 1/9/07, James P. Gunderson <jgunders@...> wrote:

> However, if I program the distance calculation incorrectly (perhaps I calculate the manhattan > distance instead of the euclidian) how does the analyzing program 'know' that I coded the
> wrong function?

I agree that this a difficult problem.

One thing that a tool could do is give some examples of calling the
function with boundary conditions or perhaps with special cases (like
0 or -3 or null) that you, the programmer, did not think of. For
simple cases, like your distance between two points example, the tool
could show some examples like

 a = new Point(0, 0);
 b = new Point(3, 4);
 assertEquals(5, a.distanceTo(b));

and have the developer review the cases it generates. This would get
tedious quite quickly for methods with complex arguments though.

Kevin

www.junitfactory.com
You send us code. We send you tests. For free.

[Ole]

Wow - That's a really well worded and thought out
description of the whole thing.  I'm impressed.

> I was
> looking at the gap between the intent of the
> programmer and the function of the target software,
> and whether it is possible to bridge that gap
> without the programmer documenting the intent.

(I'm probably saying what you are saying again here,
plus less elegantly, but I'm a work in progress)

With respect to intent I would think the following.

Given the method body and the datatypes, we can deduce
the intent 100% (Meaning that we can say precisely
what will happen per the corner cases, edge cases,
etc, however we now have to be able to say whether the
deduced intent matches the intent of the designer.

And the only way to do that 100% is to validate
whether the corner cases and edge conditions produce
the right output.  If it's a mathematical formula it
could be doing lots of rounding, summing, adding, more
rounding it could be that I would have to put a range
on the expected results.  I also need to know what the
expected result should be for all the combinations of
corner and edge conditions that constitute 100%
testing.

So what I mean by 100% method testing is that the
tests can be generated and be exhaustive for the
inputs corresponding calculated outputs that are of
interest and need to be reviewed.  That's all a
machine can do.  

Actually validating the generated output, or the
intent behind the design of the target system, in the
original case a method, is left to us.

I could change the the original question, which was
"What does it mean to test a method 100%" to a goal
like:
Have the programmer do 0 work to find out what the
characteristics of the system are and minimal work in
validating those charachteristics.

Cheers,
- Ole



--- "James P. Gunderson" <jgunders@...>
wrote:

> Kevin:
>    I totally agree that there are a number of
> reasons for developing and running tests. I think
> that your list is a great initial classification. I
> guess my point was focused on the suggestion that
> the programmer writes some code, and without
> providing any additional information, presses the
> button that generates all the test cases. JB brought
> up the question of how much one program can 'reason'
> about the behavior of another program. I agree with
> your observation that the ability to characterize a
> programs behavior is frequently needed and useful. I
> also agree that one package can do a lot to analyse
> another.
>
> All I was pointing out is that the analyzing program
> only has the source code of the target software to
> work from. Yes, a lot can be done to make sure that
> that target program has no errors in it, but it is
> harder (without extra information) to assess whether
> the target code is doing the right thing.  For
> example, suppose I write a function to calculate the
> distance between to points in space. The analysis
> can make sure that corner cases are covered, the
> code has no unnecessary complexity, that all the
> code is reachable, and so forth. And this is really
> good, useful, and provides value to me the
> programmer.  However, if I program the distance
> calculation incorrectly (perhaps I calculate the
> manhattan distance instead of the euclidian) how
> does the analyzing program 'know' that I coded the
> wrong function?  Without the addition of
> annotations, invariants, or the inclusion of test
> data, the analyzing program is hobbled. However, if
> the programmer is required to add the additional
> information, this violates the notion that the
> programmer has to do zero addtional work.
>
> Now, I would be happy to have a support system that
> provides all the benefits you outlined, and I would
> be perfectly happy to have a system that would take
> a small amount of additional info and demonstrate
> (dare we say prove) that the code I wrote is both
> error free and does the job I want it to. I was
> looking at the gap between the intent of the
> programmer and the function of the target software,
> and whether it is possible to bridge that gap
> without the programmer documenting the intent.
>
> Jim
>
>
> +++++
> Kevin Lawrence wrote:
> There are lots of reasons why one might want to have
> tests. Off the
> top of my head,
>
> - to drive the design
> - to document the design
> - to demonstrate correctness
> - to provide a safety net when changing unfamiliar
> code
>
> This latter case, which Michael Feathers talks a
> great deal about in
> WELC, is probably one of the most common reasons
> that people want
> automated tests.
>
> Putting it another way,
>
>   if I change this line of code, what other behavior
> will change will
> change as a result
>
> We ask ourselves this question every time we
> maintain unfamiliar code
> in an unfamiliar system without tests. The most
> conscientious
> developers write tests before changing that line of
> code. Most don't.
>
> This is such a common activity in software
> development that a program
> that could characterize the behavior of another
> program and report
> when the behavior has changed would be useful
> wouldn't it?
>
> As to JB's observation about whether it's possible
> to generate such
> tests for non-trivial programs - it's still early
> days for JUnit
> Factory but I think you'd be surprised by what's
> already possible.
>
>
> Kevin
>
> www.junitfactory.com
> You send us code. We send you tests. For free.
>
>
>
>
>
> [Non-text portions of this message have been
> removed]
>
>

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com 

[James P Gunderson]

Kevin:
        I think these are good ideas, although if I was deluged with 5000 test
cases that I then had to validate manually, I'd probably not use it.
This still doesn't address the issue I raised. The system would not
produce the test case:
 > a = new Point(0, 0);
 > b = new Point(3, 4);
 > assertEquals(5, a.distanceTo(b));

instead (since, in my example I coded the wrong distance function) it
would produce:

assertEquals(7, a.distance(b));

and the test would pass.  So I would have to manually check every
passing test case to make sure that it was valid. Bummer.

I would need be able to add something like:

//** Test Point(0,0), Point(3,4), 5.0 **//

as a guide to the test generator as to what was expected. Then, in
addition to all the other valuable verification tests of corner cases,
etc.; it could generate the validation test.  In our lexicon,
verification tests demonstrate that whatever the software is doing, it
is doing correctly. Validation tests confirm that the software is doing
the correct thing.  Using both, we confirm that it is both doing the
right thing, and doing it correctly.  Automated test generation is
excellent for verification, the question becomes how to we generate
validation tests as well, without overloading the programmer.

Perhaps, at this point, it is sufficient to automate the verification
tests, and leave the validation up to the developer. Possibly, with a
structured requirements document, we could extract data for validation,
and automate that as well?

Just rambling,
Jim

Kevin Lawrence wrote:

>
>
> On 1/9/07, James P. Gunderson <jgunders@...
> <mailto:jgunders%40gamma-two.com>> wrote:
>
>  > However, if I program the distance calculation incorrectly (perhaps I
> calculate the manhattan > distance instead of the euclidian) how does
> the analyzing program 'know' that I coded the
>  > wrong function?
>
> I agree that this a difficult problem.
>
> One thing that a tool could do is give some examples of calling the
> function with boundary conditions or perhaps with special cases (like
> 0 or -3 or null) that you, the programmer, did not think of. For
> simple cases, like your distance between two points example, the tool
> could show some examples like
>
> a = new Point(0, 0);
> b = new Point(3, 4);
> assertEquals(5, a.distanceTo(b));
>
> and have the developer review the cases it generates. This would get
> tedious quite quickly for methods with complex arguments though.
>
> Kevin
>
> www.junitfactory.com
> You send us code. We send you tests. For free.
>
>
>
> __________ NOD32 1969 (20070110) Information __________
>
> This message was checked by NOD32 antivirus system.
> http://www.eset.com

--
James P. Gunderson, Ph.D.                Cognitive Systems Architect
jgunders@...                   Gamma Two, Inc.
www.gamma-two.com             303.778.7400
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The only certainty is that nothing is certain.
-- Pliny the Elder, Historia Naturalis II

[Kevin Lawrence]

On 1/9/07, Greg Irvine <greg.irvine@...> wrote:
> Kevin, I think you've missed James' point, at least in the snippet of his
> email you've quoted.

I think I understood his point and agreed with it. I was trying to
make an additional point.

>
> He's saying that generating tests from existing code has the risk that the
> code doesn't do what it was intended but a test is generated to pass based
> on the incorrect code.  So you would have 100% test passes, which is
> incorrect against the actual expected results against the requirements.
>

I am trying to separate the various goals of testing and investigate
distinct strategies that would achieve each goal.

One goal is to prevent regressions: show that, after a change, the
code still does what it did before or at least show what behaviors
changed (regardless of whether the code was previously 'correct'). I
think we are agreed that a machine could do this in principle even if
it is difficult in practice. It still requires a reviewer, though, to
decide which changes were intended and which were not .


Another goal is to demonstrate correctness: demonstrate that the code
behaves correctly for all inputs. This one is not even theoretically
possible for a machine to do. But a machine can make the job of a
human reviewer easier by showing combinations of inputs and outputs
that are 'likely' to represent special or border cases.

For example, consider this method on a Triangle class:

public boolean isIsosceles() {
    return a == b || a == c || b == c;
}

A machine might generate tests:

    public void testIsIsosceles() throws Throwable {
        boolean result = new Triangle(59, 100, 59).isIsosceles();
        assertTrue("result", result);
    }

    public void testIsIsosceles1() throws Throwable {
        boolean result = new Triangle(100, 59, 59).isIsosceles();
        assertTrue("result", result);
    }

    public void testIsIsosceles2() throws Throwable {
        boolean result = new Triangle(30, 30, 30).isIsosceles();
        assertTrue("result", result);
    }

A reviewer might look at these tests and exclaim "That's not correct!
The last one is equilateral" and amend the spec to exclude equilateral
triangles from the definition of isosceles.

A second attempt might amend the code to,

  public boolean isIsosceles() {
    return a == b || a == c || b == c;
  }

The existing tests would then serve as a regression suite. After the
change, one of the tests (the previously incorrect one) would now fail
and (we hope) the others would still pass.

Assuming the tests are essentially 'free' and that we can trust the
machine to generate tests that are at least as good as they were
before, we can throw all the existing tests away and generate new
ones.

Then we might get a test like this,

    public void testIsIsosceles2() throws Throwable {
        boolean result = new Triangle(36, 36, 36).isIsosceles();
        assertFalse("result", result);
    }

and be happy that our method now behaves correctly.

There are a lot of 'if's and 'assuming's in all that but, with a
machine that could generate perfect regression tests in zero time, we
might think quite differently about the value of tests and how we use
them.

As JB suggested, it is impossible for a program to prove the
correctness of another program. Even with a human/machine combo it is
difficult and tedious which is why I still write all of my code test
first with TDD.

TDD provides excellent regression coverage too but, if it's
essentially free to generate tests to fill the gaps in my TDD
coverage, why wouldn't I?

The more astute reader will have guessed that the examples above were
lifted directly from the demo page at
http://www.junitfactory.com/demo/   The service is still experimental
and somewhat less than perfect, but I'd be interested in getting
feedback from the kind of people who think a lot about these kinds of
things. I can send an invitation to anyone who wants one.

Kevin

http://www.junitfactory.com
You send us code. We send you tests. For free.

[Kevin Lawrence]

On 1/10/07, James P. Gunderson <jgunders@...> wrote:
> Kevin:
>         I think these are good ideas, although if I was deluged with 5000 test
> cases that I then had to validate manually, I'd probably not use it.

Our posts crossed in the ether but I hope it was clear that I agree
with you. We, at Agitar, have put a lot of effort into the
verification scenario but we have not neglected the verification
scenario either. We are tackling it on a number of fronts.

One approach is similar to the approach you suggested where you seed
the test generator with some known-to-be-correct cases. The problem
with this approach is that, in the limit, it becomes easier to write
the test manually than to prod the machine into finding the right
case.  If the machine can take these suggestions and riff on them to
find 'cases like them' it becomes more interesting though.

Another approach is one that David Saff describes in his "The Practice
of Theories" (http://www.tiny.cc/HttFz) paper in which the human
proposes hypotheses and the test generator finds examples that confirm
the hypothesis or that prove it to be false.

Yet another approach - my favorite - is to have the machine generate
lots of examples of input-output sets and make it easy for a human
reader to review them and spot inconsistencies. Finding smart ways of
filtering down all the possible examples and smart ways of presenting
them to the user are the two main challenges in this approach.

Meanwhile, we think there is value in pure regression testing too.

Kevin

http://www.junitfactory.com
You send us code. We send you tests. For free.

[Ole]

Just a comment on this part:


>This one is not
>even theoretically
>possible for a machine to do.
 
>For example, consider this method on a Triangle
>class:
 
> public boolean isIsosceles() {
>     return a == b || a == c || b == c;
> }
>

Suppose a,b,c are int's.

Further, this is a triangle so

0 < [a,b,c]

is always true.

This is an invarient constraint of the model for the
triangle.

A good design tool should ask questions like that with

respect to the defined parameters.

Then to complete enumerate the test, we define the
following points:

aMax bMax cMax
aMin bMin cMin

aMin-1 bMin-1 cMin-1
aMax+1 bMax+1 cMax+1

In range means a legal java int that respects the
the invarient constraint.

aInRange bInRange cInRange ExpectedReturnValue
   2        2        2          true
   2        2        3          true
   2        3        3          true
   3        2        2          true
   3        3        2          true
   3        3        3          true
   1        2        3          false
   3        2        1          false
   2        3        1          false


Then you could do the same process using
aMin, aMin=1, bMin...etc.
finally forming the picture of all the points
that are interesting.

With respect to this,

if we were to add aMin bMin cMin to the
table above, we would be adding

   1        1        1

This falls in the same pattern as the above
points, but the tester might find it interesting/
comforting just to check that point.

Sam goes for aMax, bMax, cMax

Then for aMin-1 and aMax+1...these should
never be allowed to enter the test...an exception
should be thrown.  Thus for the enumerated set of
these
possibilities, an exception is always thrown, so
that's what the test tests for.


So these points represent the 100% test.

It's possible to create tests using different
datapoints, but they'll follow the same pattern.

So if these tests pass, everything should pass.

Cheers,
- Ole











--- Kevin Lawrence <kev.lawrence@...> wrote:

> On 1/9/07, Greg Irvine <greg.irvine@...>
> wrote:
> > Kevin, I think you've missed James' point, at
> least in the snippet of his
> > email you've quoted.
>
> I think I understood his point and agreed with it. I
> was trying to
> make an additional point.
>
> >
> > He's saying that generating tests from existing
> code has the risk that the
> > code doesn't do what it was intended but a test is
> generated to pass based
> > on the incorrect code.  So you would have 100%
> test passes, which is
> > incorrect against the actual expected results
> against the requirements.
> >
>
> I am trying to separate the various goals of testing
> and investigate
> distinct strategies that would achieve each goal.
>
> One goal is to prevent regressions: show that, after
> a change, the
> code still does what it did before or at least show
> what behaviors
> changed (regardless of whether the code was
> previously 'correct'). I
> think we are agreed that a machine could do this in
> principle even if
> it is difficult in practice. It still requires a
> reviewer, though, to
> decide which changes were intended and which were
> not .
>
>
> Another goal is to demonstrate correctness:
> demonstrate that the code
> behaves correctly for all inputs. This one is not
> even theoretically
> possible for a machine to do. But a machine can make
> the job of a
> human reviewer easier by showing combinations of
> inputs and outputs
> that are 'likely' to represent special or border
> cases.
>
> For example, consider this method on a Triangle
> class:
>
> public boolean isIsosceles() {
>     return a == b || a == c || b == c;
> }
>
> A machine might generate tests:
>
>     public void testIsIsosceles() throws Throwable {
>         boolean result = new Triangle(59, 100,
> 59).isIsosceles();
>         assertTrue("result", result);
>     }
>
>     public void testIsIsosceles1() throws Throwable
> {
>         boolean result = new Triangle(100, 59,
> 59).isIsosceles();
>         assertTrue("result", result);
>     }
>
>     public void testIsIsosceles2() throws Throwable
> {
>         boolean result = new Triangle(30, 30,
> 30).isIsosceles();
>         assertTrue("result", result);
>     }
>
> A reviewer might look at these tests and exclaim
> "That's not correct!
> The last one is equilateral" and amend the spec to
> exclude equilateral
> triangles from the definition of isosceles.
>
> A second attempt might amend the code to,
>
>   public boolean isIsosceles() {
>     return a == b || a == c || b == c;
>   }
>
> The existing tests would then serve as a regression
> suite. After the
> change, one of the tests (the previously incorrect
> one) would now fail
> and (we hope) the others would still pass.
>
> Assuming the tests are essentially 'free' and that
> we can trust the
> machine to generate tests that are at least as good
> as they were
> before, we can throw all the existing tests away and
> generate new
> ones.
>
> Then we might get a test like this,
>
>     public void testIsIsosceles2() throws Throwable
> {
>         boolean result = new Triangle(36, 36,
> 36).isIsosceles();
>         assertFalse("result", result);
>     }
>
> and be happy that our method now behaves correctly.
>
> There are a lot of 'if's and 'assuming's in all that
> but, with a
> machine that could generate perfect regression tests
> in zero time, we
> might think quite differently about the value of
> tests and how we use
> them.
>
> As JB suggested, it is impossible for a program to
> prove the
> correctness of another program. Even with a
> human/machine combo it is
> difficult and tedious which is why I still write all
> of my code test
> first with TDD.
>
> TDD provides excellent regression coverage too but,
> if it's
> essentially free to generate tests to fill the gaps
> in my TDD
> coverage, why wouldn't I?
>
> The more astute reader will have guessed that the
> examples above were
> lifted directly from the demo page at
> http://www.junitfactory.com/demo/   The service is
> still experimental
> and somewhat less than perfect, but I'd be
> interested in getting
> feedback from the kind of people who think a lot
> about these kinds of
> things. I can send an invitation to anyone who wants
> one.
>
> Kevin
>
> http://www.junitfactory.com
> You send us code. We send you tests. For free.
>

 
____________________________________________________________________________________
Need a quick answer? Get one in minutes from people who know.
Ask your question on www.Answers.yahoo.com

[Kevin Lawrence]

On 1/10/07, Ole Ersoy <ole_ersoy@...> wrote:

> Just a comment on this part:
>
>
> >This one is not
> >even theoretically
> >possible for a machine to do.
>
> >For example, consider this method on a Triangle
> >class:
>
> > public boolean isIsosceles() {
> >     return a == b || a == c || b == c;
> > }
> >
>
> aInRange bInRange cInRange ExpectedReturnValue
>    2        2        2          true
>    2        2        3          true
>    2        3        3          true
>    3        2        2          true
>    3        3        2          true
>    3        3        3          true
>    1        2        3          false
>    3        2        1          false
>    2        3        1          false
>

> So these points represent the 100% test.

But the point that several other people have made is that it is
impossible for the machine to know whether these tests are correct. If
the code is wrong, the test will be wrong too.

Kevin

[Ole]

Sure - This assumes the code is correct and then
verifies that the enumerated set of inputs
maps to the set of outputs.

This is represents the maximum amount of help that can
be provided to the tester.

That's why I started the discussion with:

"What does 100% method unit testing mean?"

What is the max we can take to.  This is the max.

Some of the data points I mentioned can be left out,
and still fit my definition of 100%, but if more of
them are left out, then the code is not as well test
as it could be with automation support.

Cheers,
- Ole



--- Kevin Lawrence <kev.lawrence@...> wrote:

> On 1/10/07, Ole Ersoy <ole_ersoy@...> wrote:
> > Just a comment on this part:
> >
> >
> > >This one is not
> > >even theoretically
> > >possible for a machine to do.
> >
> > >For example, consider this method on a Triangle
> > >class:
> >
> > > public boolean isIsosceles() {
> > >     return a == b || a == c || b == c;
> > > }
> > >
> >
> > aInRange bInRange cInRange ExpectedReturnValue
> >    2        2        2          true
> >    2        2        3          true
> >    2        3        3          true
> >    3        2        2          true
> >    3        3        2          true
> >    3        3        3          true
> >    1        2        3          false
> >    3        2        1          false
> >    2        3        1          false
> >
>
> > So these points represent the 100% test.
>
> But the point that several other people have made is
> that it is
> impossible for the machine to know whether these
> tests are correct. If
> the code is wrong, the test will be wrong too.
>
> Kevin
>

 
____________________________________________________________________________________
Yahoo! Music Unlimited
Access over 1 million songs.
http://music.yahoo.com/unlimited

[Ole]

So I guess we could summarize it by saying:

Assuming the code is correct, it's possible to
generate the "100% enumerated" set of tests.

However this does not necessarily match the set that
one would run if the code really is correct.

Thus it's up to the tester to verify that the
generated set is exhaustive.  The goal of the tester
is to make sure that this set is exhaustive.

Cheers,
- Ole


--- Ole Ersoy <ole_ersoy@...> wrote:

> Sure - This assumes the code is correct and then
> verifies that the enumerated set of inputs
> maps to the set of outputs.
>
> This is represents the maximum amount of help that
> can
> be provided to the tester.
>
> That's why I started the discussion with:
>
> "What does 100% method unit testing mean?"
>
> What is the max we can take to.  This is the max.
>
> Some of the data points I mentioned can be left out,
> and still fit my definition of 100%, but if more of
> them are left out, then the code is not as well test
> as it could be with automation support.
>
> Cheers,
> - Ole
>
>
>
> --- Kevin Lawrence <kev.lawrence@...> wrote:
>
> > On 1/10/07, Ole Ersoy <ole_ersoy@...> wrote:
> > > Just a comment on this part:
> > >
> > >
> > > >This one is not
> > > >even theoretically
> > > >possible for a machine to do.
> > >
> > > >For example, consider this method on a Triangle
> > > >class:
> > >
> > > > public boolean isIsosceles() {
> > > >     return a == b || a == c || b == c;
> > > > }
> > > >
> > >
> > > aInRange bInRange cInRange ExpectedReturnValue
> > >    2        2        2          true
> > >    2        2        3          true
> > >    2        3        3          true
> > >    3        2        2          true
> > >    3        3        2          true
> > >    3        3        3          true
> > >    1        2        3          false
> > >    3        2        1          false
> > >    2        3        1          false
> > >
> >
> > > So these points represent the 100% test.
> >
> > But the point that several other people have made
> is
> > that it is
> > impossible for the machine to know whether these
> > tests are correct. If
> > the code is wrong, the test will be wrong too.
> >
> > Kevin
> >
>
>
>
>  
>

____________________________________________________________________________________
> Yahoo! Music Unlimited
> Access over 1 million songs.
> http://music.yahoo.com/unlimited
>


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com 

[dsaff]

On 1/10/07, James P. Gunderson <jgunders@...> wrote:

> So I would have to manually check every
> passing test case to make sure that it was valid. Bummer.
>
> I would need be able to add something like:
>
> //** Test Point(0,0), Point(3,4), 5.0 **//
>
> as a guide to the test generator as to what was expected. Then, in
> addition to all the other valuable verification tests of corner cases,
> etc.; it could generate the validation test.

I find myself often tempted to do something like this.  But then, I
wonder if this is much better than

@Test public void pythagoreanDistance() {
  assertEquals(5.0, new Point(0.0).distanceTo(new Point 3,4));
}

As I see it, the advantages of generation from comments, as you're
proposing are:
G1) The validation code sits close to the implementation code, making
it easier to look at both and reason about them
G2) Specific behavior examples can sit next to general statements
about correct behavior, with the same syntax
G3) A custom domain-specific language can reduce typing.

The advantages of the JUnit approach are:
J1) Java is actually usually a pretty good language for describing the
behavior of Java programs (this is not entirely tongue-in-cheek.  For
example, bash is a rather poor language with which to describe the
behavior of shell scripts)
J2) When it's hard to write Java to validate my implementation code,
sometimes that means my code has a bad design.
J3) Not having to learn a custom domain-specific language reduces cognitive load

I currently believe that it's easier to improve tooling around a
JUnit-like framework to get the advantages of code generation than
vice-versa.  I think that G1, looking at tests and implementation
simultaneously, is something that IDE's can still improve upon
tremendously when using JUnit.  Ideally, when a test failed, I would
automatically be able to see the failure log, the test, and the
implementation code in side-by-side windows.  My Theories work is
designed to bring the G2 advantages to people using JUnit, allowing
general statements to be written and evaluated in a JUnit style.  As
to G3, I've yet to see a domain-specific language (DSL) for tests that
really excites me*, but I believe that someday I might.

None of this touches on the idea of generating characterization tests,
as with junitfactory.  What I'm getting at is the best way for a
developer to write down what the implementation code should do, to get
automated support for verifying that it does.

Thanks,

   David Saff

* (esoteric side comment) especially since I'm a firm believer that
every external DSL maintained long enough eventually becomes a poorly
implemented general-purpose OO language.  (See ant)

[Ole]

I would think though that if the tool
produced 5000 test cases to be manually
checked, this is a great indication
that the method should be refactored
to reduce the number of possible combinations
of input and output values that are of interest.

For instance with the pythogorean theorom
example we are only interested in one case
where sideA and sideB have legal values and
and the cases when they do not.

So that's
(1) sideA > 0 && side B>0
(2) sidaA <=0
(3) sideB <=0

I could also do sideA <= AMax...etc.

That's all I care about.

If (1) works, I'm good to go as far as the method is
concerned.

So I would expect the tool to point out to me
that (1) is the one that has legal values
and the others are the ones where we are testing
what happens when illegal values are passed.

(I'm switching terminonology a little from (1) = type
(1) testing)

So if the type (1) tests are greater than say 10,
then this could be an indication that the method
should be broken down further.

This fits well with the notion that testing is about
good design.

Good design is easy to validate by the developer.

Another cool thing I would like to see is a
visualization of how the test generator came up with
the combinations that it suggests should be tested.

This could be a series of sequence diagrams showing
how
each input value travels through the method to produce
the output, as well as a tree diagram showing how all
the combinations for input and output were derived.

Also, if there were a ceiling on the number of number
of combinations per method that could be tested, this
would address the concern by some that computational
time could be an issue for methods that produced a lot
of combinations.  The tool would stop as soon as it
reach the ceiling and suggest that the method be
broken down further.

Cheers,
Ole



--- "James P. Gunderson" <jgunders@...>
wrote:

> Kevin:
> I think these are good ideas, although if I was
> deluged with 5000 test
> cases that I then had to validate manually, I'd
> probably not use it.
> This still doesn't address the issue I raised. The
> system would not
> produce the test case:
>  > a = new Point(0, 0);
>  > b = new Point(3, 4);
>  > assertEquals(5, a.distanceTo(b));
>
> instead (since, in my example I coded the wrong
> distance function) it
> would produce:
>
> assertEquals(7, a.distance(b));
>
> and the test would pass.  So I would have to
> manually check every
> passing test case to make sure that it was valid.
> Bummer.
>
> I would need be able to add something like:
>
> //** Test Point(0,0), Point(3,4), 5.0 **//
>
> as a guide to the test generator as to what was
> expected. Then, in
> addition to all the other valuable verification
> tests of corner cases,
> etc.; it could generate the validation test.  In our
> lexicon,
> verification tests demonstrate that whatever the
> software is doing, it
> is doing correctly. Validation tests confirm that
> the software is doing
> the correct thing.  Using both, we confirm that it
> is both doing the
> right thing, and doing it correctly.  Automated test
> generation is
> excellent for verification, the question becomes how
> to we generate
> validation tests as well, without overloading the
> programmer.
>
> Perhaps, at this point, it is sufficient to automate
> the verification
> tests, and leave the validation up to the developer.
> Possibly, with a
> structured requirements document, we could extract
> data for validation,
> and automate that as well?
>
> Just rambling,
> Jim
>
> Kevin Lawrence wrote:
> >
> >
> > On 1/9/07, James P. Gunderson
> <jgunders@...
> > <mailto:jgunders%40gamma-two.com>> wrote:
> >
> >  > However, if I program the distance calculation
> incorrectly (perhaps I
> > calculate the manhattan > distance instead of the
> euclidian) how does
> > the analyzing program 'know' that I coded the
> >  > wrong function?
> >
> > I agree that this a difficult problem.
> >
> > One thing that a tool could do is give some
> examples of calling the
> > function with boundary conditions or perhaps with
> special cases (like
> > 0 or -3 or null) that you, the programmer, did not
> think of. For
> > simple cases, like your distance between two
> points example, the tool
> > could show some examples like
> >
> > a = new Point(0, 0);
> > b = new Point(3, 4);
> > assertEquals(5, a.distanceTo(b));
> >
> > and have the developer review the cases it
> generates. This would get
> > tedious quite quickly for methods with complex
> arguments though.
> >
> > Kevin
> >
> > www.junitfactory.com
> > You send us code. We send you tests. For free.
> >
> >
> >
> > __________ NOD32 1969 (20070110) Information
> __________
> >
> > This message was checked by NOD32 antivirus
> system.
> > http://www.eset.com
>
> --
> James P. Gunderson, Ph.D.                Cognitive
> Systems Architect
> jgunders@...                   Gamma Two,
> Inc.
> www.gamma-two.com             303.778.7400
>

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> The only certainty is that nothing is certain.
> -- Pliny the Elder, Historia Naturalis II
>



 
____________________________________________________________________________________
Want to start your own business?
Learn how on Yahoo! Small Business.
http://smallbusiness.yahoo.com/r-index

[James P Gunderson]

David:
        When I envision a system that would extract my guide, and build the
test case into the junit test set for the class I am driven by laziness.
  I totally agree that in the simplest cases, the difference between the
guide and a full test case is minimal as you pointed out.  But as the
necessary setup and preconfiguration grows in complexity I'd like to be
able to simply specify a set of inputs and a known correct result, then
have the computer do the work.

I really believe in testing, and I always write test cases, but (oh, the
shame) I don't really like to spend my time writing tests. So anything
that can reduce the hurdles between me and fully tested (both verified
and validated) is a big win.

I clearly agree with all the benefits you pointed out for generating
tests from some structured comment format in the code itself. I agree
with many of the benefits of writing the tests as java/junit code
directly.  We are not big on test driven design so the benefits of J2
are marginal; but we are strongly into test driven development. As a
result, being able to quickly and easily specify inputs and outputs for
test cases and hang them next to the implementation code would work
really well for our development processes, provided that we can then
extract those guidances into fully functional test cases.  So in the
spirit of the original 'and then the programmer presses a button, and
all the tests are generated automatically', a combination of
characterization tests for verification, and developer inserted
validation tests seems like it might cover all the bases.  Clearly this
is somewhat antithetical to the test driven design paradigm.

I share your concern about domain specific languages so perhaps the
notion would be to simply write the assert into the comments (tagged in
a way that the test generator can find it) and then let something like
junitfactory generate the test:

//** assertEquals(5.0, new Point(0.0).distanceTo(new Point 3,4)); **//

This is in java/junit, it is clean, it resides in the source code, and
(with the right tool) whatever extra is needed for the setup can be
handled by the generator. So it would seem to get all the benefits you
listed, except that tricky J2.  I guess my extreme laziness rose to the
fore. My first thought was that  by always specifying the inputs and the
output, I could avoid the repeated typing of assertEquals(...), and then
I figured that, since the comment is lexically associated with the
function being tested, there was no need to repeatedly type
distanceTo(...), so all that was left was the parameter list and the
expected result.

However, this is the slippery slope that leads to first, an overly
specific DSL; and then (apparently) inevitably a clunky general purpose
language that can be more trouble than it is worth. But it sure would
make my life easier!

Thanks for your insight,
Jim

David Saff wrote:

>
>
> On 1/10/07, James P. Gunderson <jgunders@...
> <mailto:jgunders%40gamma-two.com>> wrote:
>  > So I would have to manually check every
>  > passing test case to make sure that it was valid. Bummer.
>  >
>  > I would need be able to add something like:
>  >
>  > //** Test Point(0,0), Point(3,4), 5.0 **//
>  >
>  > as a guide to the test generator as to what was expected. Then, in
>  > addition to all the other valuable verification tests of corner cases,
>  > etc.; it could generate the validation test.
>
> I find myself often tempted to do something like this. But then, I
> wonder if this is much better than
>
> @Test public void pythagoreanDistance() {
> assertEquals(5.0, new Point(0.0).distanceTo(new Point 3,4));
> }
>
> As I see it, the advantages of generation from comments, as you're
> proposing are:
> G1) The validation code sits close to the implementation code, making
> it easier to look at both and reason about them
> G2) Specific behavior examples can sit next to general statements
> about correct behavior, with the same syntax
> G3) A custom domain-specific language can reduce typing.
>
> The advantages of the JUnit approach are:
> J1) Java is actually usually a pretty good language for describing the
> behavior of Java programs (this is not entirely tongue-in-cheek. For
> example, bash is a rather poor language with which to describe the
> behavior of shell scripts)
> J2) When it's hard to write Java to validate my implementation code,
> sometimes that means my code has a bad design.
> J3) Not having to learn a custom domain-specific language reduces
> cognitive load
>
> I currently believe that it's easier to improve tooling around a
> JUnit-like framework to get the advantages of code generation than
> vice-versa. I think that G1, looking at tests and implementation
> simultaneously, is something that IDE's can still improve upon
> tremendously when using JUnit. Ideally, when a test failed, I would
> automatically be able to see the failure log, the test, and the
> implementation code in side-by-side windows. My Theories work is
> designed to bring the G2 advantages to people using JUnit, allowing
> general statements to be written and evaluated in a JUnit style. As
> to G3, I've yet to see a domain-specific language (DSL) for tests that
> really excites me*, but I believe that someday I might.
>
> None of this touches on the idea of generating characterization tests,
> as with junitfactory. What I'm getting at is the best way for a
> developer to write down what the implementation code should do, to get
> automated support for verifying that it does.
>
> Thanks,
>
> David Saff
>
> * (esoteric side comment) especially since I'm a firm believer that
> every external DSL maintained long enough eventually becomes a poorly
> implemented general-purpose OO language. (See ant)
>
>
>

--
James P. Gunderson, Ph.D.                Cognitive Systems Architect
jgunders@...                   Gamma Two, Inc.
www.gamma-two.com             303.778.7400
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
"A computer is like an Old Testament god, with a lot of rules and no
mercy." -- Joseph Campbell

[James P Gunderson]

Ole:

Ole Ersoy wrote:
 >
 >
 > I would think though that if the tool
 > produced 5000 test cases to be manually
 > checked, this is a great indication
 > that the method should be refactored
 > to reduce the number of possible combinations
 > of input and output values that are of interest.

        Well, you are probably right, if a single method had 5000 test cases,
it might be a little overly complex.  But, in the spirit of our mythical
programmer who writes her code and then presses the 'Generate Tests'
button; it would not be unlikely for the entire package to have 5000
tests generated. This is great - we want lots of tests - provided we
don't have to go through all the passing tests to see which are
correctly testing invalid code. Clearly she is going to do incremental
build and test, but all those tests have to be looked at eventually.
That is why I proposed the guidance to tell the test generator 'I don't
care what I actually wrote, this is what the code is supposed to do.'
With the inclusion of that kind of guidance, the 100% testing with zero
(programmer) effort may be possible.

 > This fits well with the notion that testing is about
 > good design.
 >
 > Good design is easy to validate by the developer.

On an academic note, are you sure that "good design is easy to
validate"?  I've certainly been in some animated discussions in which it
was hard to reach agreement :-)

Thanks,
Jim
--
James P. Gunderson, Ph.D.                Cognitive Systems Architect
jgunders@...                   Gamma Two, Inc.
www.gamma-two.com             303.778.7400
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
"A computer is like an Old Testament god, with a lot of rules and no
mercy." -- Joseph Campbell