100%% Method Testing and Generating Unit Tests

[Ole]

Hey Guys,

I wrote an additional suggestion earlier
on the generation of unit tests, but I can't respond
to it because it has not passed through moderation
yet.

Just wanted to add this additional note:

One thing that is useful to start with with is

"What does it mean for a method to be 100% unit
tested?"

For instance suppose there's a method

public int multiply(int a, int b)

return aXb;

Do we unit test this?

Probably not.

The assumption here is that unless this always works
conceptually, as long as we pass valid integer
parameters.

So to guarantee the method all that is needed is to
check the parameters to make sure they are valid for
the runtime context.

Although we could still have a unit test on it just in
case there was an issue with the VM running the code,
the hardware, etc.  Then if there was a production
environment oddity occuring, we could run it, just to
make sure something is not wacked.

What about

public String append(String a, String b)

return a+b;

This is 100% tested too, assuming all valid java
strings are ok.  If we need to narrow the set of
Strings that are allowed, then we use constraints on
the parameters again.  Then we could test that what we
expect to happen given disallowed parameters happens,
and what we expect given permitted parameters happens.

This stuff is sort of obvious, but I figured I'd throw
it into the mix.

Happy New Year All,
- Ole





 

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com 

[dsaff]

Ole,

I'm getting my head around your suggestions.  (At least) one thing is
tripping me up:

On 12/31/06, Ole Ersoy <ole_ersoy@...> wrote:

> "What does it mean for a method to be 100% unit
> tested?"
>
> For instance suppose there's a method
>
> public int multiply(int a, int b)
>
> return aXb;
>
> Do we unit test this?

Do we write this in the first place?  Perhaps another example would
help me more.

   David Saff

[bria526]

--- In junit@..., Ole Ersoy <ole_ersoy@...> wrote:
{clip}
> What about
>
> public String append(String a, String b)
>
> return a+b;
>
> This is 100% tested too, assuming all valid java
> strings are ok.

Ole, forgive me if I'm missing your point, but I'm curious why,
assuming you even have a valid reason to write the above util method,
it is not worthy of a few tests.

Granted, I'm a bit of a perfectionist sometimes, but the above to me
begs at least boundary tests, particularly "null tests".  What would
you expect this to do when given nulls?  It wasn't obvious to me, so I
wrote a few tests (and, honestly, was surprised by the outcome):

@RunWith(Parameterized.class)
public class StringUtilTest {
  class StringUtil {
        String append(String a, String b) {
                return a + b;
        }
  }

  private String first;
  private String second;
  private String expected;
       
  @Parameters
  public static Collection data() {
       return Arrays.asList(new Object[][]{
            {null, null, "nullnull"},
            {null, "2", "null2"},
            {"1", null, "1null"},
            {"", "", ""},
            {"1", "2", "12"},
        });
  }
 
  public StringUtilTest(String first, String second, String expected) {
        this.first = first;
        this.second = second;
        this.expected = expected;
  }
       
  @Test public void Append(){
        assertEquals(expected, new StringUtil().append(first, second));
  }
}


Is this, in your opinion, overkill [and paranoid :-)]?

Cheers...
--MB

[Ole]

Hey David,

I'm going to go over some obvious stuff, which you
already know, just for clearity.

I tried to start with something really simple that
represents everything we do when coding, which is
mapping inputs to outputs and storing those inputs and
outputs in tighthy little slots on objects.

So in that example we have two slots.

Let me just define a slot as something that can
contain everything allowed/understood by a computer.

Then we constrained the slots by making them int.

Now they can only contain the subset of all values
allowed that are int.

So now we apply an Operation to the slots and put the
result in a different slot.

So we have a Class that with private int a,b,c
and

an Operation defined in
Java

public int multiply(int a, int b)
return aXb;

It does not really matter what that Operation is, the
only thing we care about is that it correctly maps the
inputs to the outputs.

So lets start with the inputs.

They have already been constrained.

But is it done or is it possible that we need to
constrain variable b further, to satisfy a API
contract definition.

For instance b must be between 100 and 200.

Quick side note(If an EMF model contained all the
variables, a, b, c, etc.) and we needed to place
constraints on a int datatype like this, we could use
an invarient constraint for that.

OK - Back to the logic.

So we constrain b further so that  100 < b < 200.

So now we are wondering whether our method works?

By looking at the constraint for a&b, being that "a"
is an int, and "b" is a further constrained int,

and the mapping the method performs we know all the
possible return values.

We also know all the possible input values.

But we don't want to test every single input to
output...

101X1, 102X1, ...

So which ones do we test?

Let's start with the assuming that were always going
to pass legal values.

So we want to test the type constraint on a and b,
being int.

So lets just assume that on the platform we are
running legal values are

-2222222222222222222 to 2222222222222222222

So we use these two end of range values.

And we test

101 and 199 for b, since that represents b's set of
edge points.

So the combinations we want to test for any
mathematical operations with these two parameters are:

 b              a
101 X -2222222222222222222
101 X 2222222222222222222
199 X -2222222222222222222
199 X 2222222222222222222

If X works the way it's supposed to, then these tests
should pass.

Also, we test
 b              a
99  X -2222222222222222222
99  X 2222222222222222222
201 X -2222222222222222222
201 X 2222222222222222222

These should not pass.

Then we assume that all other values are taken care
due to the definition of X.

So were did we get the

99,201,199,101,2222222222222222222,-2222222222222222222
values from.

Lets think of a and b as being datatypes.

b is just a new datatype that can only
have values between 100 and 200.

So a DataType is just meta data defining an allowed
set of values...which is where EMF or Java reflection
comes in.  You would use them to analyze the meta data
defined for a type.  Then combine that with knowledge
of operators to generate test cases.

Does that make more sense?

Cheers,
- Ole










--- David Saff <saff@...> wrote:

> Ole,
>
> I'm getting my head around your suggestions.  (At
> least) one thing is
> tripping me up:
>
> On 12/31/06, Ole Ersoy <ole_ersoy@...> wrote:
> > "What does it mean for a method to be 100% unit
> > tested?"
> >
> > For instance suppose there's a method
> >
> > public int multiply(int a, int b)
> >
> > return aXb;
> >
> > Do we unit test this?
>
> Do we write this in the first place?  Perhaps
> another example would
> help me more.
>
>    David Saff
>

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com 

[dsaff]

Ole,

I'm not sure I'm any closer to the main point.  Let me try to
summarize what you're suggesting:

* We would like to have methods that are 100% tested
* Getting to 100% tested takes less time if we generate tests from a
model than if we write them by hand.

Is that a fair summary?

   David Saff

On 1/2/07, Ole Ersoy <ole_ersoy@...> wrote:

> Hey David,
>
> I'm going to go over some obvious stuff, which you
> already know, just for clearity.
>
> I tried to start with something really simple that
> represents everything we do when coding, which is
> mapping inputs to outputs and storing those inputs and
> outputs in tighthy little slots on objects.
>
> So in that example we have two slots.
>
> Let me just define a slot as something that can
> contain everything allowed/understood by a computer.
>
> Then we constrained the slots by making them int.
>
> Now they can only contain the subset of all values
> allowed that are int.
>
> So now we apply an Operation to the slots and put the
> result in a different slot.
>
> So we have a Class that with private int a,b,c
> and
>
> an Operation defined in
> Java
>
> public int multiply(int a, int b)
> return aXb;
>
> It does not really matter what that Operation is, the
> only thing we care about is that it correctly maps the
> inputs to the outputs.
>
> So lets start with the inputs.
>
> They have already been constrained.
>
> But is it done or is it possible that we need to
> constrain variable b further, to satisfy a API
> contract definition.
>
> For instance b must be between 100 and 200.
>
> Quick side note(If an EMF model contained all the
> variables, a, b, c, etc.) and we needed to place
> constraints on a int datatype like this, we could use
> an invarient constraint for that.
>
> OK - Back to the logic.
>
> So we constrain b further so that  100 < b < 200.
>
> So now we are wondering whether our method works?
>
> By looking at the constraint for a&b, being that "a"
> is an int, and "b" is a further constrained int,
>
> and the mapping the method performs we know all the
> possible return values.
>
> We also know all the possible input values.
>
> But we don't want to test every single input to
> output...
>
> 101X1, 102X1, ...
>
> So which ones do we test?
>
> Let's start with the assuming that were always going
> to pass legal values.
>
> So we want to test the type constraint on a and b,
> being int.
>
> So lets just assume that on the platform we are
> running legal values are
>
> -2222222222222222222 to 2222222222222222222
>
> So we use these two end of range values.
>
> And we test
>
> 101 and 199 for b, since that represents b's set of
> edge points.
>
> So the combinations we want to test for any
> mathematical operations with these two parameters are:
>
>  b              a
> 101 X -2222222222222222222
> 101 X 2222222222222222222
> 199 X -2222222222222222222
> 199 X 2222222222222222222
>
> If X works the way it's supposed to, then these tests
> should pass.
>
> Also, we test
>  b              a
> 99  X -2222222222222222222
> 99  X 2222222222222222222
> 201 X -2222222222222222222
> 201 X 2222222222222222222
>
> These should not pass.
>
> Then we assume that all other values are taken care
> due to the definition of X.
>
> So were did we get the
>
> 99,201,199,101,2222222222222222222,-2222222222222222222
> values from.
>
> Lets think of a and b as being datatypes.
>
> b is just a new datatype that can only
> have values between 100 and 200.
>
> So a DataType is just meta data defining an allowed
> set of values...which is where EMF or Java reflection
> comes in.  You would use them to analyze the meta data
> defined for a type.  Then combine that with knowledge
> of operators to generate test cases.
>
> Does that make more sense?
>
> Cheers,
> - Ole
>
>
>
>
>
>
>
>
>
>
> --- David Saff <saff@...> wrote:
>
> > Ole,
> >
> > I'm getting my head around your suggestions.  (At
> > least) one thing is
> > tripping me up:
> >
> > On 12/31/06, Ole Ersoy <ole_ersoy@...> wrote:
> > > "What does it mean for a method to be 100% unit
> > > tested?"
> > >
> > > For instance suppose there's a method
> > >
> > > public int multiply(int a, int b)
> > >
> > > return aXb;
> > >
> > > Do we unit test this?
> >
> > Do we write this in the first place?  Perhaps
> > another example would
> > help me more.
> >
> >    David Saff
> >
>
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
>
>
>
> Yahoo! Groups Links
>
>
>
>

[Ole]

Hey David,


--- David Saff <saff@...> wrote:

> Ole,
>
> I'm not sure I'm any closer to the main point.  Let
> me try to
> summarize what you're suggesting:
>
> * We would like to have methods that are 100% tested



I like it - Although let me add that it is really
important to agree on what 100% means in a well
structured / unambiguous context.




> * Getting to 100% tested takes less time if we
> generate tests from a
> model than if we write them by hand.


Sure - getting methods that are 100% tested should
take 0 time (For the person doing the coding).

I'm really wanting to write more for elaboration, but
I'll hold off, since I'd most likely be reframing some
of the earlier material.

Please let me know if you want me to put a different
angle on it.  

Cheers,
- Ole

>
> Is that a fair summary?
>
>    David Saff
>
> On 1/2/07, Ole Ersoy <ole_ersoy@...> wrote:
> > Hey David,
> >
> > I'm going to go over some obvious stuff, which you
> > already know, just for clearity.
> >
> > I tried to start with something really simple that
> > represents everything we do when coding, which is
> > mapping inputs to outputs and storing those inputs
> and
> > outputs in tighthy little slots on objects.
> >
> > So in that example we have two slots.
> >
> > Let me just define a slot as something that can
> > contain everything allowed/understood by a
> computer.
> >
> > Then we constrained the slots by making them int.
> >
> > Now they can only contain the subset of all values
> > allowed that are int.
> >
> > So now we apply an Operation to the slots and put
> the
> > result in a different slot.
> >
> > So we have a Class that with private int a,b,c
> > and
> >
> > an Operation defined in
> > Java
> >
> > public int multiply(int a, int b)
> > return aXb;
> >
> > It does not really matter what that Operation is,
> the
> > only thing we care about is that it correctly maps
> the
> > inputs to the outputs.
> >
> > So lets start with the inputs.
> >
> > They have already been constrained.
> >
> > But is it done or is it possible that we need to
> > constrain variable b further, to satisfy a API
> > contract definition.
> >
> > For instance b must be between 100 and 200.
> >
> > Quick side note(If an EMF model contained all the
> > variables, a, b, c, etc.) and we needed to place
> > constraints on a int datatype like this, we could
> use
> > an invarient constraint for that.
> >
> > OK - Back to the logic.
> >
> > So we constrain b further so that  100 < b < 200.
> >
> > So now we are wondering whether our method works?
> >
> > By looking at the constraint for a&b, being that
> "a"
> > is an int, and "b" is a further constrained int,
> >
> > and the mapping the method performs we know all
> the
> > possible return values.
> >
> > We also know all the possible input values.
> >
> > But we don't want to test every single input to
> > output...
> >
> > 101X1, 102X1, ...
> >
> > So which ones do we test?
> >
> > Let's start with the assuming that were always
> going
> > to pass legal values.
> >
> > So we want to test the type constraint on a and b,
> > being int.
> >
> > So lets just assume that on the platform we are
> > running legal values are
> >
> > -2222222222222222222 to 2222222222222222222
> >
> > So we use these two end of range values.
> >
> > And we test
> >
> > 101 and 199 for b, since that represents b's set
> of
> > edge points.
> >
> > So the combinations we want to test for any
> > mathematical operations with these two parameters
> are:
> >
> >  b              a
> > 101 X -2222222222222222222
> > 101 X 2222222222222222222
> > 199 X -2222222222222222222
> > 199 X 2222222222222222222
> >
> > If X works the way it's supposed to, then these
> tests
> > should pass.
> >
> > Also, we test
> >  b              a
> > 99  X -2222222222222222222
> > 99  X 2222222222222222222
> > 201 X -2222222222222222222
> > 201 X 2222222222222222222
> >
> > These should not pass.
> >
> > Then we assume that all other values are taken
> care
> > due to the definition of X.
> >
> > So were did we get the
> >
> >
>

99,201,199,101,2222222222222222222,-2222222222222222222

> > values from.
> >
> > Lets think of a and b as being datatypes.
> >
> > b is just a new datatype that can only
> > have values between 100 and 200.
> >
> > So a DataType is just meta data defining an
> allowed
> > set of values...which is where EMF or Java
> reflection
> > comes in.  You would use them to analyze the meta
> data
> > defined for a type.  Then combine that with
> knowledge
> > of operators to generate test cases.
> >
> > Does that make more sense?
> >
> > Cheers,
> > - Ole
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > --- David Saff <saff@...> wrote:
> >
> > > Ole,
> > >
> > > I'm getting my head around your suggestions.
> (At
> > > least) one thing is
> > > tripping me up:
> > >
> > > On 12/31/06, Ole Ersoy <ole_ersoy@...>
> wrote:
> > > > "What does it mean for a method to be 100%
> unit
> > > > tested?"
> > > >
> > > > For instance suppose there's a method
> > > >
> > > > public int multiply(int a, int b)
> > > >
> > > > return aXb;
> > > >
> > > > Do we unit test this?
> > >
>

=== message truncated ===


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com 

[Cédric Beust ♔]

On 1/2/07, Ole Ersoy <ole_ersoy@...> wrote:
>
>
> > * Getting to 100% tested takes less time if we
> > generate tests from a
> > model than if we write them by hand.
>
>
> Sure - getting methods that are 100% tested should
> take 0 time (For the person doing the coding).


That's a very deceptive statement.

Not only is it very hard to reach 100% testing (just because your code
coverage claims 100% doesn't mean you are covering 100% of your code), but
it's actually harmful to try to reach this goal.  There is a point of
diminishing return where you are better off improving your application that
shooting for this elusive 100% testing.

--
Cédric
http://testng.org


[Non-text portions of this message have been removed]

[bria526]

OIe --

Just curious if you had any comment on my question to you earlier in the
chain.  Thanks!

--MB


[Non-text portions of this message have been removed]

[Ole]

We first have to agree on what 100% means.

I provided a very simple example in the earlier post.

The example gives a 100% tested tested method example,
given that the assumptions hold.

This example can be entirely generated using DataType
meta data.

Thus it's an immidiate result of using a modeling
approach to generating a solution / code.

It could be implemented by reading the defined
Operation into a model defining the additional meta
data for the Java DataTypes and then running a test
generator.

EMF already has a Java Parser for generating Ecore
Models from Annotated java code.

So this could be used.

My challenge to you now is describing unambiguously
how the point of diminishing return is reached through
a series of simple examples.







--- Cédric Beust ♔  <cbeust@...> wrote:

> On 1/2/07, Ole Ersoy <ole_ersoy@...> wrote:
> >
> >
> > > * Getting to 100% tested takes less time if we
> > > generate tests from a
> > > model than if we write them by hand.
> >
> >
> > Sure - getting methods that are 100% tested should
> > take 0 time (For the person doing the coding).
>
>
> That's a very deceptive statement.
>
> Not only is it very hard to reach 100% testing (just
> because your code
> coverage claims 100% doesn't mean you are covering
> 100% of your code), but
> it's actually harmful to try to reach this goal.
> There is a point of
> diminishing return where you are better off
> improving your application that
> shooting for this elusive 100% testing.
>
> --
> Cédric
> http://testng.org
>
>
> [Non-text portions of this message have been
> removed]
>
>

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com 

[nails762]

Ole Ersoy wrote:

>  > * Getting to 100% tested takes less time if we
>  > generate tests from a
>  > model than if we write them by hand.
>
> Sure - getting methods that are 100% tested should
> take 0 time (For the person doing the coding).

Do you mean "it would be nice if 100% tested code took 0 time"? or that
"it should be possible today to have 100% tested code in 0 time"?

It seems to me the first is not possible and the second is obviously
false. What am I missing?
--
J. B. (Joe) Rainsberger :: http://www.jbrains.ca
Your guide to software craftsmanship
JUnit Recipes: Practical Methods for Programmer Testing
2005 Gordon Pask Award for contribution Agile Software Practice

[Ole]

Mike,

Do you mean the part about writing the code first, and
then testing or writing the test first and then
coding?

Let me just suppose that that is it.

Here's what I do.

When ever I'm working with a new API,
I start with a JUnit test.

Then I define what I want to do in a little cookbook
item like this:

Challenge

     Load Maven Pom File

Solution

     See Discussion (Because it's not really a quicky)

Discussion

     First Create the ResourceSet

     ResourceSet resourceSet = new ResourceSetImpl();

     Then create a URI pointing to the object

     URI uri = ....





Then I take the process / solution
that I'm trying to validate that works
and put it in the JUnit test.

Then I run the test.

And keep tweaking until I get it right.

As I tweak I make updates to the cookbook,
just so I can remember what I did :-)

A test is code and code is code.

I like the test first concept because
it encourages you to break the step you are doing
into a few lines that are easy to test and asks
you to validate your thinking.  By doing this you have
confidence in your code.

That said it's nice when your tests verify your code
continuosly.

So with what I did above it's possible that I tested
my "Challenge" using JUnit,

then copied and pasted the code into a method, and
this method also contained other code.

Now I broke my ability to continously test that
"section" of code as the .java file is updated...It's
possible that someone applies a patch that overwrites
it...the test still passes, but the actual code is
broken...

But that's an easy thing to fix...just mentioned it
because it's important...

Cheers,
- Ole



--- Mike Bria <bria526xp@...> wrote:

> OIe --
>
> Just curious if you had any comment on my question
> to you earlier in the
> chain.  Thanks!
>
> --MB
>
>
> [Non-text portions of this message have been
> removed]
>
>

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com 

[Ole]

To answer that you would have to
analyze the simple example I gave earlier
and then show why you think what you just said
is accurate.


--- "J. B. Rainsberger" <jbrains762@...> wrote:

> Ole Ersoy wrote:
>
> >  > * Getting to 100% tested takes less time if we
> >  > generate tests from a
> >  > model than if we write them by hand.
> >
> > Sure - getting methods that are 100% tested should
> > take 0 time (For the person doing the coding).
>
> Do you mean "it would be nice if 100% tested code
> took 0 time"? or that
> "it should be possible today to have 100% tested
> code in 0 time"?
>
> It seems to me the first is not possible and the
> second is obviously
> false. What am I missing?
> --
> J. B. (Joe) Rainsberger :: http://www.jbrains.ca
> Your guide to software craftsmanship
> JUnit Recipes: Practical Methods for Programmer
> Testing
> 2005 Gordon Pask Award for contribution Agile
> Software Practice
>

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com 

[Cédric Beust ♔]

On 1/3/07, Ole Ersoy <ole_ersoy@...> wrote:

>
> We first have to agree on what 100% means.
>
> I provided a very simple example in the earlier post.
>
> The example gives a 100% tested tested method example,
> given that the assumptions hold.
>
> This example can be entirely generated using DataType
> meta data.

Yes, but your example is too simple since it only contains one branch (most
examples extolling the virtues of code coverage have this flaw :-)).

100% coverage means you should cover all the possible branches of your code,
which is impossible to achieve in practice.  Even when code coverage tools
report 100% coverage (hardly ever happens in my experience), they are still
lying to you.

Consider the following simple code:

public URL createUrl(int a, int b) {

try {
  if (a == 0 || b == 0) {
  }

  if (c == 0 && d == 0) {
  }
}
catch(IOException ex) {
}
catch(MalformedURLException ex) {
}

100% coverage for this code means something like 2*2*2*2 = 16 test cases
(I'm approximating, but you get the idea).

Do you really feel it's useful to write 16 test cases for these 6 lines of
code?

Wouldn't you be better off black-box testing this function instead, and
ignore the screams of complaint from your code coverage tool?

--
Cédric
http://testng.org


[Non-text portions of this message have been removed]

[bria526]

On 1/3/07, Cédric Beust ♔ <cbeust@...> wrote:
{snip}

>  Consider the following simple code:
>
>  public URL createUrl(int a, int b) {
>
>  try {
>    if (a == 0 || b == 0) {
>    }
>
>  if (c == 0 && d == 0) {
>    }
>  }
>  catch(IOException ex) {
>  }
>  catch(MalformedURLException ex) {
>  }
>
>Do you really feel it's useful to write 16 test cases for these 6

lines of code?

Cedric --

I don't quite understand your example.  Why would that code ever be
written?  It doesn't do anything at all (at least as written).

I assume there is implied code that has been omitted to save
keystrokes and eye movement?

If in that case I were to be imaginative and dream up some [also
simple] additional code that would, at the least, birth c, d, and/or a
URL or give rise to the possibility of the two stated exceptions, then
yes, to me this method appears to have enough margin for error that I
would want it protected with unit tests.

Did I miss your point?

Cheers...
--MB

[Cédric Beust ♔]

Mike,

Yes, I think you missed my point :-)

I was just showing a snippet of code with a few branches.  I'm sure you can
easily find similar code in your own applications (two ifs, two booleans in
each if and a try/catch).

Each branch multiplies the number of code paths by two, and that's not even
taking exceptions into account, which add multipliers of their own.

The bottom line is that in average, I would expect that a typical Java
method probably requires 10-20 test cases to be 100% branch covered, hence
my belief that trying to achieve this number is not just a bad idea, it's
unprofessional :-)

--
Cédric
http://testng.org


On 1/3/07, Mike Bria <bria526xp@...> wrote:

>
> On 1/3/07, Cédric Beust ♔ <cbeust@...> wrote:
> {snip}
> >  Consider the following simple code:
> >
> >  public URL createUrl(int a, int b) {
> >
> >  try {
> >    if (a == 0 || b == 0) {
> >    }
> >
> >  if (c == 0 && d == 0) {
> >    }
> >  }
> >  catch(IOException ex) {
> >  }
> >  catch(MalformedURLException ex) {
> >  }
> >
> >Do you really feel it's useful to write 16 test cases for these 6
> lines of code?
>
> Cedric --
>
> I don't quite understand your example.  Why would that code ever be
> written?  It doesn't do anything at all (at least as written).
>
> I assume there is implied code that has been omitted to save
> keystrokes and eye movement?
>
> If in that case I were to be imaginative and dream up some [also
> simple] additional code that would, at the least, birth c, d, and/or a
> URL or give rise to the possibility of the two stated exceptions, then
> yes, to me this method appears to have enough margin for error that I
> would want it protected with unit tests.
>
> Did I miss your point?
>
> Cheers...
> --MB
>
>
>
> Yahoo! Groups Links
>
>
>
>

[Non-text portions of this message have been removed]

[Ole]

Cédric,

Excellent example old Chap.

So a branch is something that takes the code down a
certain path.

if ( a > 10 )

is a branch.

I think we agree that it's possible to enumerate the
entire set of branches.

So for instance

a > 10

a < 10

a = 10

represents 3 different paths.

And if I could switch it up
with something like this instead:

a > 10 && b < 5

a < 10

a = 10

I could enumerate the test set like I did for my
simple example by
assuming edge points / the max allowable values for a
and b (Incidentally in my simple example I did not
test values greater than the max allowable values,
which I should have)

So now we have test input data set:

a                     b
-------------------------------------
11                    4
9                     4
10                    4
aBiggest              4
aBiggest+1            4
aSmallest             4
aSmallest-1           4

11                    5
9                     5
10                    5
aBiggest              5
aBiggest+1            5
aSmallest             5
aSmallest-1           5

11                    bMax
9                     bMax
10                    bMax
aMax                  bMax
aMax+1                bMax
aMin                  bMax
aMin-1                bMax

etc. with bMin and bMin-1

That's not so bad...

Your ipod can do that in 5 nanoseconds.

But these are just the inputs.

We also have to generate the outputs.

Which is also easy to do and we would use the
method to calculate them.  This assumes that there
are no constraints on the output.  Lets just say that
the output is just an int
c.

Suppose c can only be 1 or 2.

Well, that implies further constraints on the inputs,
so we have to go back and reconstrain.

However in this example c can be anything, so we just
calculate the possible outputs.

Then we run it through testing.

Now we could test

a = 22
a = 23

etc.

but that would be silly since we assume that
since we passed values through our enumerated branch
list, that represent the beginning and end of the
range of allowed values and even the points external
to that range, we covered everything.

So now I can look at my generated test code
and see all the inputs, the expected outputs,
and run the tests.

I personally would like this since it does a lot of
the grunt work for me.

It's possible to have a method
with a gazillion variables and paths...

but I would think that that would be a ripe case for
refactoring, because it's tricky to test :-)

As a matter of fact...it would be nice to have
something that enumerates the branches and generates a
report on which methods has a number of branches over
a certain limit...for code refactoring and maintenance
purposes.

Cheers,
- Ole






--- Cédric Beust ♔  <cbeust@...> wrote:

> On 1/3/07, Ole Ersoy <ole_ersoy@...> wrote:
> >
> > We first have to agree on what 100% means.
> >
> > I provided a very simple example in the earlier
> post.
> >
> > The example gives a 100% tested tested method
> example,
> > given that the assumptions hold.
> >
> > This example can be entirely generated using
> DataType
> > meta data.
>
>
> Yes, but your example is too simple since it only
> contains one branch (most
> examples extolling the virtues of code coverage have
> this flaw :-)).
>
> 100% coverage means you should cover all the
> possible branches of your code,
> which is impossible to achieve in practice.  Even
> when code coverage tools
> report 100% coverage (hardly ever happens in my
> experience), they are still
> lying to you.
>
> Consider the following simple code:
>
> public URL createUrl(int a, int b) {
>
> try {
>   if (a == 0 || b == 0) {
>   }
>
>   if (c == 0 && d == 0) {
>   }
> }
> catch(IOException ex) {
> }
> catch(MalformedURLException ex) {
> }
>
> 100% coverage for this code means something like
> 2*2*2*2 = 16 test cases
> (I'm approximating, but you get the idea).
>
> Do you really feel it's useful to write 16 test
> cases for these 6 lines of
> code?
>
> Wouldn't you be better off black-box testing this
> function instead, and
> ignore the screams of complaint from your code
> coverage tool?
>
> --
> Cédric
> http://testng.org
>
>
> [Non-text portions of this message have been
> removed]
>
>

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com 

[Jeff Langr]

Quoting Cédric Beust ?  <cbeust@...>:

> public URL createUrl(int a, int b) {
> try {
>   if (a == 0 || b == 0) {
>   }
>   if (c == 0 && d == 0) {
>   }
> }
> catch(IOException ex) {
> }
> catch(MalformedURLException ex) {
> }
>
> 100% coverage for this code means something like 2*2*2*2 = 16 test cases
> (I'm approximating, but you get the idea).
>
> Do you really feel it's useful to write 16 test cases for these 6 lines of
> code?

Greetings Cedric,

I prefer not to use coverage tools (in fact, I strongly resist  
characterizing the quality of unit testing with coverage numbers), but  
instead test-drive as much as possible in the code. In other words,  
the complex conditionals don't get complex until I have a second test  
that drives out the "or" part of the clause. So in theory I'd end up  
with the combinatorial explosion you're suggesting.

However, in practice that doesn't happen often.

Sometimes these complex conditionals end up as method calls:
   if (a == null || b == null)
becomes:
   if (areAnyNull(a, b)) // yes this is a trivial example, but it's  
not atypical

I end up with comprehensive tests against areAnyNull; that's easy to  
test (it often becomes a public method on another class). Then I only  
concern myself with two cases to cover the "if" block in the enclosing  
method (createUrl in this example). I often find that encapsulating  
the complex conditionals within separate methods (a) helps make the  
code easier to follow,  (b) often helps reduce the overall code size  
(it makes some duplication more obvious), and (c) makes some defects  
very apparent.

Even if I need to explode the tests, they end up extremely short,  
sometimes a single line, by virtue of test refactoring.

In general, it's the refactoring that keeps the tests in TDD sane. I  
don't end up with a lot of methods that have more than a couple of  
branch statements, so I don't usually have a problem with covering all  
the combinations.

Regardless, I'd rather have the test that demonstrates that the rhs of  
the conditional really works, and that all of the combinations thus  
really work. Particularly given short circuit evaluation, I've  
encountered many cases where not completely testing this allowed  
defects that weren't found until much later.

Regards,
Jeff

[Cédric Beust ♔]

Ole,

I'm not sure what point you are trying to make with this example, and it
seems to me we drifted away from the point that I was making:  100% branch
covering in your tests is an impossible goal to achieve, even if my iPod can
test ten lines of Java in 5 nanoseconds.

Hence my recommendation to let the developer exercise their judgment where
to draw the line, and in deciding where they should cover a few branches as
opposed to black-box testing their code.

--
Cédric
http://testng.org


On 1/3/07, Ole Ersoy <ole_ersoy@...> wrote:

>
> CÃ(c)dric,
>
> Excellent example old Chap.
>
> So a branch is something that takes the code down a
> certain path.
>
> if ( a > 10 )
>
> is a branch.
>
> I think we agree that it's possible to enumerate the
> entire set of branches.
>
> So for instance
>
> a > 10
>
> a < 10
>
> a = 10
>
> represents 3 different paths.
>
> And if I could switch it up
> with something like this instead:
>
> a > 10 && b < 5
>
> a < 10
>
> a = 10
>
> I could enumerate the test set like I did for my
> simple example by
> assuming edge points / the max allowable values for a
> and b (Incidentally in my simple example I did not
> test values greater than the max allowable values,
> which I should have)
>
> So now we have test input data set:
>
> a                     b
> -------------------------------------
> 11                    4
> 9                     4
> 10                    4
> aBiggest              4
> aBiggest+1            4
> aSmallest             4
> aSmallest-1           4
>
> 11                    5
> 9                     5
> 10                    5
> aBiggest              5
> aBiggest+1            5
> aSmallest             5
> aSmallest-1           5
>
> 11                    bMax
> 9                     bMax
> 10                    bMax
> aMax                  bMax
> aMax+1                bMax
> aMin                  bMax
> aMin-1                bMax
>
> etc. with bMin and bMin-1
>
> That's not so bad...
>
> Your ipod can do that in 5 nanoseconds.
>
> But these are just the inputs.
>
> We also have to generate the outputs.
>
> Which is also easy to do and we would use the
> method to calculate them.  This assumes that there
> are no constraints on the output.  Lets just say that
> the output is just an int
> c.
>
> Suppose c can only be 1 or 2.
>
> Well, that implies further constraints on the inputs,
> so we have to go back and reconstrain.
>
> However in this example c can be anything, so we just
> calculate the possible outputs.
>
> Then we run it through testing.
>
> Now we could test
>
> a = 22
> a = 23
>
> etc.
>
> but that would be silly since we assume that
> since we passed values through our enumerated branch
> list, that represent the beginning and end of the
> range of allowed values and even the points external
> to that range, we covered everything.
>
> So now I can look at my generated test code
> and see all the inputs, the expected outputs,
> and run the tests.
>
> I personally would like this since it does a lot of
> the grunt work for me.
>
> It's possible to have a method
> with a gazillion variables and paths...
>
> but I would think that that would be a ripe case for
> refactoring, because it's tricky to test :-)
>
> As a matter of fact...it would be nice to have
> something that enumerates the branches and generates a
> report on which methods has a number of branches over
> a certain limit...for code refactoring and maintenance
> purposes.
>
> Cheers,
> - Ole
>
>
>
>
>
>
> --- CÃ(c)dric Beust â™"  <cbeust@...> wrote:
>
> > On 1/3/07, Ole Ersoy <ole_ersoy@...> wrote:
> > >
> > > We first have to agree on what 100% means.
> > >
> > > I provided a very simple example in the earlier
> > post.
> > >
> > > The example gives a 100% tested tested method
> > example,
> > > given that the assumptions hold.
> > >
> > > This example can be entirely generated using
> > DataType
> > > meta data.
> >
> >
> > Yes, but your example is too simple since it only
> > contains one branch (most
> > examples extolling the virtues of code coverage have
> > this flaw :-)).
> >
> > 100% coverage means you should cover all the
> > possible branches of your code,
> > which is impossible to achieve in practice.  Even
> > when code coverage tools
> > report 100% coverage (hardly ever happens in my
> > experience), they are still
> > lying to you.
> >
> > Consider the following simple code:
> >
> > public URL createUrl(int a, int b) {
> >
> > try {
> >   if (a == 0 || b == 0) {
> >   }
> >
> >   if (c == 0 && d == 0) {
> >   }
> > }
> > catch(IOException ex) {
> > }
> > catch(MalformedURLException ex) {
> > }
> >
> > 100% coverage for this code means something like
> > 2*2*2*2 = 16 test cases
> > (I'm approximating, but you get the idea).
> >
> > Do you really feel it's useful to write 16 test
> > cases for these 6 lines of
> > code?
> >
> > Wouldn't you be better off black-box testing this
> > function instead, and
> > ignore the screams of complaint from your code
> > coverage tool?
> >
> > --
> > Cédric
> > http://testng.org
> >
> >
> > [Non-text portions of this message have been
> > removed]
> >
> >
>
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
>
>
>
> Yahoo! Groups Links
>
>
>
>

[Non-text portions of this message have been removed]



 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/junit/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/junit/join
    (Yahoo! ID required)

<*> To change settings via email:
    mailto:junit-digest@...
    mailto:junit-fullfeatured@...

<*> To unsubscribe from this group, send an email to:
    junit-unsubscribe@...

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/
 

[Ole]

Cédric,

Just building on your example to show how 100% testing
would be done.

It's easy to automate, and would be a valuable tool in
the developer toolbelt.

Also you proved that it's possible to do.  I just
built on top of your proof.

Cheers,
- Ole




--- Cédric Beust ♔  <cbeust@...> wrote:

> Ole,
>
> I'm not sure what point you are trying to make with
> this example, and it
> seems to me we drifted away from the point that I
> was making:  100% branch
> covering in your tests is an impossible goal to
> achieve, even if my iPod can
> test ten lines of Java in 5 nanoseconds.
>
> Hence my recommendation to let the developer
> exercise their judgment where
> to draw the line, and in deciding where they should
> cover a few branches as
> opposed to black-box testing their code.
>
> --
> Cédric
> http://testng.org
>
>
> On 1/3/07, Ole Ersoy <ole_ersoy@...> wrote:
> >
> > CÃ(c)dric,
> >
> > Excellent example old Chap.
> >
> > So a branch is something that takes the code down
> a
> > certain path.
> >
> > if ( a > 10 )
> >
> > is a branch.
> >
> > I think we agree that it's possible to enumerate
> the
> > entire set of branches.
> >
> > So for instance
> >
> > a > 10
> >
> > a < 10
> >
> > a = 10
> >
> > represents 3 different paths.
> >
> > And if I could switch it up
> > with something like this instead:
> >
> > a > 10 && b < 5
> >
> > a < 10
> >
> > a = 10
> >
> > I could enumerate the test set like I did for my
> > simple example by
> > assuming edge points / the max allowable values
> for a
> > and b (Incidentally in my simple example I did not
> > test values greater than the max allowable values,
> > which I should have)
> >
> > So now we have test input data set:
> >
> > a                     b
> > -------------------------------------
> > 11                    4
> > 9                     4
> > 10                    4
> > aBiggest              4
> > aBiggest+1            4
> > aSmallest             4
> > aSmallest-1           4
> >
> > 11                    5
> > 9                     5
> > 10                    5
> > aBiggest              5
> > aBiggest+1            5
> > aSmallest             5
> > aSmallest-1           5
> >
> > 11                    bMax
> > 9                     bMax
> > 10                    bMax
> > aMax                  bMax
> > aMax+1                bMax
> > aMin                  bMax
> > aMin-1                bMax
> >
> > etc. with bMin and bMin-1
> >
> > That's not so bad...
> >
> > Your ipod can do that in 5 nanoseconds.
> >
> > But these are just the inputs.
> >
> > We also have to generate the outputs.
> >
> > Which is also easy to do and we would use the
> > method to calculate them.  This assumes that there
> > are no constraints on the output.  Lets just say
> that
> > the output is just an int
> > c.
> >
> > Suppose c can only be 1 or 2.
> >
> > Well, that implies further constraints on the
> inputs,
> > so we have to go back and reconstrain.
> >
> > However in this example c can be anything, so we
> just
> > calculate the possible outputs.
> >
> > Then we run it through testing.
> >
> > Now we could test
> >
> > a = 22
> > a = 23
> >
> > etc.
> >
> > but that would be silly since we assume that
> > since we passed values through our enumerated
> branch
> > list, that represent the beginning and end of the
> > range of allowed values and even the points
> external
> > to that range, we covered everything.
> >
> > So now I can look at my generated test code
> > and see all the inputs, the expected outputs,
> > and run the tests.
> >
> > I personally would like this since it does a lot
> of
> > the grunt work for me.
> >
> > It's possible to have a method
> > with a gazillion variables and paths...
> >
> > but I would think that that would be a ripe case
> for
> > refactoring, because it's tricky to test :-)
> >
> > As a matter of fact...it would be nice to have
> > something that enumerates the branches and
> generates a
> > report on which methods has a number of branches
> over
> > a certain limit...for code refactoring and
> maintenance
> > purposes.
> >
> > Cheers,
> > - Ole
> >
> >
> >
> >
> >
> >
> > --- CÃ(c)dric Beust â™"  <cbeust@...>
> wrote:
> >
> > > On 1/3/07, Ole Ersoy <ole_ersoy@...>
> wrote:
> > > >
> > > > We first have to agree on what 100% means.
> > > >
> > > > I provided a very simple example in the
> earlier
> > > post.
> > > >
> > > > The example gives a 100% tested tested method
> > > example,
> > > > given that the assumptions hold.
> > > >
> > > > This example can be entirely generated using
> > > DataType
> > > > meta data.
> > >
> > >
> > > Yes, but your example is too simple since it
> only
> > > contains one branch (most
> > > examples extolling the virtues of code coverage
> have
> > > this flaw :-)).
> > >
> > > 100% coverage means you should cover all the
> > > possible branches of your code,
> > > which is impossible to achieve in practice.
> Even
>

=== message truncated ===


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com 

[Cédric Beust ♔]

On 1/3/07, Ole Ersoy <ole_ersoy@...> wrote:
>
> CÃ(c)dric,
>
> Just building on your example to show how 100% testing
> would be done.


The question wasn't to figure out whether it can be done (it's a graph
traversal problem, we know it can be solved) but to decide whether it's
practical to try to reach 100% branch coverage.

You seem to persist in believing that it is.

It's pretty obvious to me that your tool will never scale past toy programs,
but good luck with your quest.

--
Cédric
http://testng.org


[Non-text portions of this message have been removed]