another one from oredev

[Gojko Adzic]

James Bach talked today at Oredev, i especially liked the part on
differences between testing and checking. here's a short write-up:

http://gojko.net/2009/11/06/checking-is-not-testing-testing-is-not-checking/

--
gojko adzic
http://gojko.net

[C. Titus Brown]

On Fri, Nov 06, 2009 at 11:21:03AM +0000, Gojko Adzic wrote:
> James Bach talked today at Oredev, i especially liked the part on
> differences between testing and checking. here's a short write-up:
>
> http://gojko.net/2009/11/06/checking-is-not-testing-testing-is-not-checking/

Hi Gojko,

just read this -- thanks!

Isn't his 'testing' simply 'exploratory testing'?

(I always have trouble figuring out whether such attempts to redefine
terminology are serious; they seem like a waste of time to me.   There will
always be people who use the right words in the right way but don't have a
deeper understanding, and there will always be people who use the wrong words
but do have a deeper understanding.  I aspire to have a deeper understanding &
I could care less what things are called; the understanding is the point.)

--titus
--
C. Titus Brown, ctb@...

[ligerly]

It helps to frame the conversation.  Tests are attempts to investigate
software and typically find new bugs; checks are often automated and check
that bugs have not been created in code that was working.  This really helps
a conversation about where the testing is happening in the lifecycle, often
avoiding misunderstandings (whihc seem to occur a lot talking about agile
testing).
cheers,
Erik

On Tue, Nov 10, 2009 at 2:11 PM, C. Titus Brown <t@...> wrote:

>
>
> On Fri, Nov 06, 2009 at 11:21:03AM +0000, Gojko Adzic wrote:
> > James Bach talked today at Oredev, i especially liked the part on
> > differences between testing and checking. here's a short write-up:
> >
> >
> hestttp://gojko.net/2009/11/06/checking-is-not-testing-testing-is-not-checking/<http://gojko.net/2009/11/06/checking-is-not-testing-testing-is-not-checking/>
>
> Hi Gojko,
>
> just read this -- thanks!
>
> Isn't his 'testing' simply 'exploratory testing'?
>
> (I always have trouble figuring out whether such attempts to redefine
> terminology are serious; they seem like a waste of time to me. There will
> always be people who use the right words in the right way but don't have a
> deeper understanding, and there will always be people who use the wrong
> words
> but do have a deeper understanding. I aspire to have a deeper understanding
> &
> I could care less what things are called; the understanding is the point.)
>
> --titus
> --
> C. Titus Brown, ctb@... <ctb%40msu.edu>
>  
>

[Brian Marick]

On Nov 9, 2009, at 9:30 PM, erik petersen wrote:

> It helps to frame the conversation.  Tests are attempts to  
> investigate software and typically find new bugs; checks are often  
> automated and check that bugs have not been created in code that was  
> working.

That absolutely does not capture what "checks" are. Let me give you an  
example. I'm working on an application that's used to schedule animals  
for teaching labs. An early version of the app had a drag and drop  
interface with three panels. I later changed it to a clicking  
interface with four panels. I had to rewrite the UI code and the tests/
checks of the UI code. It might seem the checks were worthless during  
the rewrite (and in 1997, I would have thought so). But it turns out  
that all (or almost all) of those checks contained an idea that needed  
to be preserved in the new interface. They were reminders of important  
things to think about, and that (it seemed to me) repaid the cost of  
rewriting them.

That was a big surprise to me.

The, um, ecosystem of programmer tests is subtle and complicated.  
Simple distinctions like "tests" and "checks" don't cut it.

-----
Brian Marick, independent consultant
Mostly on agile methods with a testing slant
Author of _Programming Cocoa with Ruby_
www.exampler.com, www.exampler.com/blog, www.twitter.com/marick

[Hubert Matthews]

Brian Marick wrote:

> That absolutely does not capture what "checks" are. Let me give you an
> example. I'm working on an application that's used to schedule animals
> for teaching labs. An early version of the app had a drag and drop
> interface with three panels. I later changed it to a clicking
> interface with four panels. I had to rewrite the UI code and the tests/
> checks of the UI code. It might seem the checks were worthless during
> the rewrite (and in 1997, I would have thought so). But it turns out
> that all (or almost all) of those checks contained an idea that needed
> to be preserved in the new interface. They were reminders of important
> things to think about, and that (it seemed to me) repaid the cost of
> rewriting them.
>
> That was a big surprise to me.

I'm not surprised by this at all.  When modelling a system I consider
types, operations and business rules.  People usually focus on the first
of these (types/classes/data), then take a good stab at the second
(operations), followed by some grudging attempt at the third
(rules/constraints).  What you probably did was put the checking of
business rules into your UI checking code, thus conflating UI checking
with acceptance checking.  Refactoring the rules into non-UI code that is
tested separately from a wafer-thin UI would be the ideal solution.

--
Hubert Matthews         http://www.oxyware.com/
Software Consultant     hubert@...

[Brian Marick]

On Nov 10, 2009, at 7:32 AM, Hubert Matthews wrote:

> What you probably did was put the checking of
> business rules into your UI checking code, thus conflating UI checking
> with acceptance checking. Refactoring the rules into non-UI code  
> that is
> tested separately from a wafer-thin UI would be the ideal solution.

No, that is not what I did. It's a model-view-presenter architecture.  
The presenters were tested. Changing the controls the presenters  
control meant changing the tests. That was not busy-work, though,  
because the ideas behind the tests served as a checklist that guided  
the code rewrite.

-----
Brian Marick, independent consultant
Mostly on agile methods with a testing slant
Author of _Programming Cocoa with Ruby_
www.exampler.com, www.exampler.com/blog, www.twitter.com/marick

[C. Titus Brown]

On Mon, Nov 09, 2009 at 10:57:38PM -0600, Brian Marick wrote:

>
> On Nov 9, 2009, at 9:30 PM, erik petersen wrote:
>
> > It helps to frame the conversation.  Tests are attempts to  
> > investigate software and typically find new bugs; checks are often  
> > automated and check that bugs have not been created in code that was  
> > working.
>
> That absolutely does not capture what "checks" are. Let me give you an  
> example. I'm working on an application that's used to schedule animals  
> for teaching labs. An early version of the app had a drag and drop  
> interface with three panels. I later changed it to a clicking  
> interface with four panels. I had to rewrite the UI code and the tests/
> checks of the UI code. It might seem the checks were worthless during  
> the rewrite (and in 1997, I would have thought so). But it turns out  
> that all (or almost all) of those checks contained an idea that needed  
> to be preserved in the new interface. They were reminders of important  
> things to think about, and that (it seemed to me) repaid the cost of  
> rewriting them.
>
> That was a big surprise to me.
>
> The, um, ecosystem of programmer tests is subtle and complicated.  
> Simple distinctions like "tests" and "checks" don't cut it.

Exactly.  I feel like attempts at precise definitions are Procrustean in nature
[0].

If the discussion highlights an interesting distinction that helps people
gain understanding, great!  But I think that should be the benchmark for
such attempts, not whether or not the definition is useful (because generally
it's not, in the long run).

--titus

[0] http://en.wikipedia.org/wiki/Procrustes -- hey, if YOU'D paid for a liberal
arts education, wouldn't you want to flaunt it?!
--
C. Titus Brown, ctb@...

[Gojko Adzic]

> just read this -- thanks!
>
> Isn't his 'testing' simply 'exploratory testing'?
>

My understanding is that Bach wanted to put more emphasis on exploratory
testing as a task that testers should be dealing with and that
"checking" is something best delegated to a machine. From that
perspective, it makes sense to try to differentiate between the two.

--
gojko adzic
http://gojko.net

[Brian Marick]

Earlier I claimed that having to rewrite tests because of UI changes  
was nevertheless worthwhile because the tests were a checklist of  
things I had to continue to make true. Here are some details: http://bit.ly/b2942

-----
Brian Marick, independent consultant
Mostly on agile methods with a testing slant
Author of _Programming Cocoa with Ruby_
www.exampler.com, www.exampler.com/blog, www.twitter.com/marick

[ligerly]

Hi Brian,
OK, how about this alternate explanation. I'll add design to what I'd
limited to testing and checking.
I think there is a fundamental switching at the heart of what you have been
doing.  As part of specification by design a design-intention-check is used
to discover that something is functioning to a design intent. Once it is
working, it typically serves as a regression-check.

If you add new aspects to your design, the regression-checks guard against
regressions in the existing design.    If you revisit the original design,
the regression-checks switch to design-intention-checks (or reminders as you
called them in your email or a frozen decision in your blog post).
This could apply equally to assert style tests (checks?) and traditional
scripted tests (checks?).  Does that make more sense?

I'm still undecided as to the value of the distinction with tests, but it
added context to some recent confusing conversations differentiating between
regression checks and tests focused on feedback driven functional and
failure discovery.  It was well understood by the developers in the Agile 09
Chicago session where Michael gave the lightning talk about it (chaired by
Jim Shore).

cheers,
Erik

On Tue, Nov 10, 2009 at 3:57 PM, Brian Marick <marick@...> wrote:

>
>
>
> On Nov 9, 2009, at 9:30 PM, erik petersen wrote:
>
> > It helps to frame the conversation. Tests are attempts to
> > investigate software and typically find new bugs; checks are often
> > automated and check that bugs have not been created in code that was
> > working.
>
> That absolutely does not capture what "checks" are. Let me give you an
> example. I'm working on an application that's used to schedule animals
> for teaching labs. An early version of the app had a drag and drop
> interface with three panels. I later changed it to a clicking
> interface with four panels. I had to rewrite the UI code and the tests/
> checks of the UI code. It might seem the checks were worthless during
> the rewrite (and in 1997, I would have thought so). But it turns out
> that all (or almost all) of those checks contained an idea that needed
> to be preserved in the new interface. They were reminders of important
> things to think about, and that (it seemed to me) repaid the cost of
> rewriting them.
>
> That was a big surprise to me.
>
> The, um, ecosystem of programmer tests is subtle and complicated.
> Simple distinctions like "tests" and "checks" don't cut it.
>
> -----
> Brian Marick, independent consultant
> Mostly on agile methods with a testing slant
> Author of _Programming Cocoa with Ruby_
> www.exampler.com, www.exampler.com/blog, www.twitter.com/marick
>
>  
>

[Brian Marick]

On Nov 12, 2009, at 4:02 AM, erik petersen wrote:

> Hi Brian,
> OK, how about this alternate explanation. [...]  Does that make more  
> sense?


I don't know. How will the world be meaningfully different if we talk  
like this versus old-fashioned lingo? What does this buy us over  
longer strings of words like "exploratory testing: needed, no matter  
how enthusiastic you are about junit, newly-test-infected programmer-
type" and "scripted manual tests: ick!" and "talking about examples of  
executing the app, using business language, before writing code:  
helpful!" and "rerunning tests frequently: more helpful than Marick  
thought in 1997, but still risky as the tests come to execute more and  
more lines of code".

The philosopher Richard Rorty said (roughly) in the introduction to /
Consequences of Pragmatism/:

        Long conversations about the definitions of words
        like "force" and "mass" have turned out to be
        wonderfully useful. Even longer conversations about
        the definitions of words like "truth" and "good"
        haven't, as it turns out. So maybe we philosophers
        should talk about something else?

I think the definitions in our field are closer to "truth" than  
"mass". Maybe that's because, like "truth", they're words wrapped up  
with people's self-image and identity.



-----
Brian Marick, independent consultant
Mostly on agile methods with a testing slant
Author of _Programming Cocoa with Ruby_
www.exampler.com, www.exampler.com/blog, www.twitter.com/marick

[Lisa Crispin-2]

I worry that terminology might be used to keep newbies on the outside of a
profession. If they don't learn our secret handshakes, we won't talk to
them. I personally have never learned all the official testing terminology,
if there is such a thing. Some people say that means I don't know anything
about testing.

OTOH, it's helpful to have a common language. I would love it if
"integration test" meant the same thing to all of us, but I know for sure
that it means different things to different people - just to give one
example.

Gerard Meszaros influenced me and Janet when we wrote our book to use
"standard terminology" (in this case, Gerard's, since we don't know any
better standard) as much as possible. So I stopped saying "integration
tests' and started saying "component tests" since that's what I meant, but
other people may not know that term either. I'm not going to beat them up if
they don't. I'll just keep trying to communicate as well as I can.

I do like it when people try to come up with clearer terminology. If it
works for a critical mass of people, it'll stick, and it'll help.
-- Lisa

On Thu, Nov 12, 2009 at 6:18 PM, Brian Marick <marick@...> wrote:

>
>
>
> On Nov 12, 2009, at 4:02 AM, erik petersen wrote:
>
> > Hi Brian,
> > OK, how about this alternate explanation. [...] Does that make more
> > sense?
>
> I don't know. How will the world be meaningfully different if we talk
> like this versus old-fashioned lingo? What does this buy us over
> longer strings of words like "exploratory testing: needed, no matter
> how enthusiastic you are about junit, newly-test-infected programmer-
> type" and "scripted manual tests: ick!" and "talking about examples of
> executing the app, using business language, before writing code:
> helpful!" and "rerunning tests frequently: more helpful than Marick
> thought in 1997, but still risky as the tests come to execute more and
> more lines of code".
>
> The philosopher Richard Rorty said (roughly) in the introduction to /
> Consequences of Pragmatism/:
>
> Long conversations about the definitions of words
> like "force" and "mass" have turned out to be
> wonderfully useful. Even longer conversations about
> the definitions of words like "truth" and "good"
> haven't, as it turns out. So maybe we philosophers
> should talk about something else?
>
> I think the definitions in our field are closer to "truth" than
> "mass". Maybe that's because, like "truth", they're words wrapped up
> with people's self-image and identity.
>
>
> -----
> Brian Marick, independent consultant
> Mostly on agile methods with a testing slant
> Author of _Programming Cocoa with Ruby_
> www.exampler.com, www.exampler.com/blog, www.twitter.com/marick
>
>  
>

--
Lisa Crispin
Co-author with Janet Gregory, _Agile Testing: A Practical Guide for Testers
and Agile Teams_ (Addison-Wesley 2009)
http://lisacrispin.com

[Brian Marick]

On Nov 12, 2009, at 6:32 PM, Lisa Crispin wrote:

> I worry that terminology might be used to keep newbies on the  
> outside of a profession. If they don't learn our secret handshakes,  
> we won't talk to them.

The thing to do when someone offers you a secret handshake is to say,  
"Oh. Um. That one. I'm afraid I've forgotten quite how that goes,  
because of, you know, the new one. Can you talk me through it?"

-----
Brian Marick, independent consultant
Mostly on agile methods with a testing slant
Author of _Programming Cocoa with Ruby_
www.exampler.com, www.exampler.com/blog, www.twitter.com/marick

[adam_peter.knight]

I agree, I feel that the testing vs checking argument, although making a
very valid point, is introducing the risk of diverging the Software
Testing interpretation of these words away from the english meaning.
This may hinder us when communicating with people outside the industry,
such as our customers, and those new to the community.  The Cambridge
dictionary definition of a test is "to do something in order to discover
if something is safe, works correctly, etc., or if something is
present". This can validly apply to an automated regression test or a
scripted test as much as an exploratory investigation. I would suggest
that assessment vs checking as components of testing would be more
accurate literally, assessment being "when you judge or decide the
amount, value, quality or importance of something, or the judgment or
decision that is made". However it is not my intention to throw further
terminology into the mix, but rather to concur with Lisa on the dangers
of imposing specific, exclusive meanings onto everyday language within
the profession.

Adam.

----------------------------------------------------------
http://www.linkedin.com/in/adampknight
http://a-sisyphean-task.blogspot.com
----------------------------------------------------------


--- In agile-testing@..., Lisa Crispin <lisa.crispin@...>
wrote:
>
> I worry that terminology might be used to keep newbies on the outside
of a
> profession. If they don't learn our secret handshakes, we won't talk
to
> them. I personally have never learned all the official testing
terminology,
> if there is such a thing. Some people say that means I don't know
anything
> about testing.
>
> OTOH, it's helpful to have a common language. I would love it if
> "integration test" meant the same thing to all of us, but I know for
sure
> that it means different things to different people - just to give one
> example.
>
> Gerard Meszaros influenced me and Janet when we wrote our book to use
> "standard terminology" (in this case, Gerard's, since we don't know
any
> better standard) as much as possible. So I stopped saying "integration
> tests' and started saying "component tests" since that's what I meant,
but
> other people may not know that term either. I'm not going to beat them
up if
> they don't. I'll just keep trying to communicate as well as I can.
>
> I do like it when people try to come up with clearer terminology. If
it

> works for a critical mass of people, it'll stick, and it'll help.
> -- Lisa
>
> On Thu, Nov 12, 2009 at 6:18 PM, Brian Marick marick@... wrote:
>
> >
> >
> >
> > On Nov 12, 2009, at 4:02 AM, erik petersen wrote:
> >
> > > Hi Brian,
> > > OK, how about this alternate explanation. [...] Does that make

more
> > > sense?
> >
> > I don't know. How will the world be meaningfully different if we
talk
> > like this versus old-fashioned lingo? What does this buy us over
> > longer strings of words like "exploratory testing: needed, no matter
> > how enthusiastic you are about junit, newly-test-infected
programmer-
> > type" and "scripted manual tests: ick!" and "talking about examples
of
> > executing the app, using business language, before writing code:
> > helpful!" and "rerunning tests frequently: more helpful than Marick
> > thought in 1997, but still risky as the tests come to execute more
and
> > more lines of code".
> >
> > The philosopher Richard Rorty said (roughly) in the introduction to
/

> > Consequences of Pragmatism/:
> >
> > Long conversations about the definitions of words
> > like "force" and "mass" have turned out to be
> > wonderfully useful. Even longer conversations about
> > the definitions of words like "truth" and "good"
> > haven't, as it turns out. So maybe we philosophers
> > should talk about something else?
> >
> > I think the definitions in our field are closer to "truth" than
> > "mass". Maybe that's because, like "truth", they're words wrapped up
> > with people's self-image and identity.
> >
> >
> > -----
> > Brian Marick, independent consultant
> > Mostly on agile methods with a testing slant
> > Author of _Programming Cocoa with Ruby_
> > www.exampler.com, www.exampler.com/blog, www.twitter.com/marick
> >
> >
> >
>
>
>
> --
> Lisa Crispin
> Co-author with Janet Gregory, _Agile Testing: A Practical Guide for

Testers
> and Agile Teams_ (Addison-Wesley 2009)
> http://lisacrispin.com
>