Nabble - Agile Testing

Re: Can we always pre-prepare automated regression tests?

2009-12-09T13:12:00Z

Also would we not expect a large portion of the segmenting of the work
to have been done at the point of breaking down the requirements into
stories prior to allocating the stories to the sprint? At this stage I
try to ensure that the stories are small enough to process in full in a
sprint. If the stories are too large to large to run through this
process in a single iteration then I would be inclined to:-

a) Use the recursive nature of stories to break the story down further
into something that can be processed in this way and put the other items
back on the backlog
b) Discuss the size of the stories with the team and Product
Manager/customer

I tend to have to do this more often than I would like due to some
issues that need ironing out elsewhere in our process.

Adam.

--- In agile-testing@..., "Matthew" <matt.heusser@...>
wrote:

>
>
>
> --- In agile-testing@..., Steven Gordon sgordonphd@ wrote
> >The question is whether it is reasonable to expect to be
> >able to go from high level acceptance criteria to first
> >cut unambiguous criteria to additional unambiguous criteria
> >discovered by exploratory testing (not to mention some
> >direct feedback from the Product Manager) all under the
> >umbrella of the same single story and get it completely
> >done in a single iteration (along with several other stories).
> >
>
> We've been doing it at Socialtext for 37 out of the past 41 two-week

iterations (with a couple long iterations around the holidays.)
>
> But we expect regression testing to spill over into the next
iteration, and estimation and planning on the next iteration start
early.
>
> Clearly, there's more than one way to do it.
>
>
>
> regards,
>
> --heusser
>

Re: Re: Please be advised

2009-12-09T10:47:55Z

I'm happy for people to use material from the book. Some major parts of the
book are based, with permission and credit, on the work of others - for
example, Brian Marick's agile testing matrix. But, in the book, we called
those the Quadrants, Brian does not. If someone wants to call them the
Quadrants, I think they should credit us with that term.

Things like 'Ten Principles for an Agile Tester" or "An Iteration in the
Life" seem like generic terms, except that I don't know of them being used
in the context of agile testing except in our book.

As I said, I know of several good sources for agile testing training that I
recommend all the time. Some of them refer to our work in their courses, as
we refer to their work in our book and courses. We all have to share
knowledge. But if it's something really specific, not just common practice
that everyone does, it needs to be denoted as such, not presented as
original material. These people are marketing a course, for money, they
aren't giving away information via a blog post or something.

The WAQB originally asked me and Janet to develop a course for them. We
declined. It looks like they decided to just use our material anyway.

I personally do not think there is an easy way to "certify" someone as an
agile tester. I do urge people to get training, I just don't know how to
measure what they can apply from their training with a multiple-choice test.

-- Lisa

On Wed, Dec 9, 2009 at 11:29 AM, Maaret <maaret@...> wrote:

>
>
> I felt that looking at the easily available evidence before asking a group
> this wide is also reasonable. I did read the blog post, the googlecached
> course outline and compared briefly to the book's table of contents. Sure,
> there's a lot of same things, but I did not see it as copypaste. Much of my
> favorite stuff from the book was excluded.
>
> I assume that in addition someone has seen their materials? Or heard an
> explanation on why the same outline that they once removed was reposted on
> the website, is that really intentional and would they assume that if
> checked once, there would no longer be checks?
>
> I still feel the request to avoid someone in training is pretty strong
> statement. It may be justified, but personally I feel I'd still like a bit
> more extensiveness on the evidence side.
>
> Maaret
>
>
> --- In agile-testing@... <agile-testing%40yahoogroups.com>,
> "Matthew" <matt.heusser@...> wrote:
> >
> >
> > --- In agile-testing@... <agile-testing%40yahoogroups.com>,
> "Maaret" <maaret@> wrote:
> > >
> > > This is a pretty strong statement with little evidence
> > >provided for the community to judge for themselves
> > >at least in my perspective.
> > >
> >
> > Hey, Asking for evidence is reasonable.
> >
> > My blog post ( http://www.stpcollaboarative.com/matt ) has a link to the
> googlecached course outline as presented by the WAQB, which is a direct
> cut/paste of the outline from Lisa and Janet's book. It is provided without
> Attribution or permission; that is a copyright violation.
> >
> > Check lisa's twitter posts ( http://twitter.com/lisacrispin ); this is
> the /second/ time they've done this. In other words, they cut/pasted the
> outline before, were asked to put it down, waited six months, and put it
> back up again.
> >
> > Well, it's almost a cut/paste. Most likely, it's a re-type, as there are
> typographical errors.
> >
> > Do your own research. Draw your own conclusions.
> >
> >
> >
> > regards,
> >
> >
> > Matthew Heusser
> > Blog: http://blogs.stpcollaborative.com/matt
> >
>
>
>

--
Lisa Crispin
Co-author with Janet Gregory, _Agile Testing: A Practical Guide for Testers
and Agile Teams_ (Addison-Wesley 2009)
http://lisacrispin.com

Re: Please be advised

2009-12-09T10:42:36Z

I understand that, but shouldn't any vendor do their due diligence of
courses they market?

Nobody at either WAQB or Bouvet has agreed to send me the course materials
so I can verify that they are not using material from our book without
proper credit.

I give Bouvet the benefit of the doubt, but it would be hard to recommend
them while being unsure of their competence to judge the courses they
market.
-- Lisa

On Wed, Dec 9, 2009 at 7:16 AM, Dave Rooney <dave.rooney@...> wrote:

>
>
> Just to be clear, Bouvet != WAQB. Bouvet was having Steen Lerche-Jensen
> of WAQB put on the Agile Testing course.
>
> Dave Rooney
> Co-founder and Consultant, The Agile Consortium
> "Maximizing the value of your IT investments!"
> E-mail: dave@... <dave%40theagileconsortium.com>
> Twitter: daverooneyca
> http://www.theagileconsortium.com
> http://practicalagility.blogspot.com
>
>
> Lisa Crispin wrote:
> >
> >
> > A company called Bouvet, alternatively called the "World Agile
> > Qualifications Board", is marketing an agile testing course that is a
> > direct rip-off of the book written by me and Janet, as well as our own
> > course.
> >
> > If you are interested in agile testing training, there are plenty of
> > legitimate sources of this, including our course, Elisabeth
> > Hendrickson, Dale Emery, and I believe, ethical companies such as SQE
> > and Rally, just to name a few. Don't give any business to people who
> > don't actually know anything, but simply steal material from
> > practitioners.
> > Thanks,
> > Lisa
> >
> > --
> > Lisa Crispin
> > Co-author with Janet Gregory, _Agile Testing: A Practical Guide for
> > Testers and Agile Teams_ (Addison-Wesley 2009)
> > http://lisacrispin.com
> >
>
>
>

--
Lisa Crispin
Co-author with Janet Gregory, _Agile Testing: A Practical Guide for Testers
and Agile Teams_ (Addison-Wesley 2009)
http://lisacrispin.com

Re: Please be advised

2009-12-09T10:29:20Z

I felt that looking at the easily available evidence before asking a group this wide is also reasonable. I did read the blog post, the googlecached course outline and compared briefly to the book's table of contents. Sure, there's a lot of same things, but I did not see it as copypaste. Much of my favorite stuff from the book was excluded.

I assume that in addition someone has seen their materials? Or heard an explanation on why the same outline that they once removed was reposted on the website, is that really intentional and would they assume that if checked once, there would no longer be checks?

I still feel the request to avoid someone in training is pretty strong statement. It may be justified, but personally I feel I'd still like a bit more extensiveness on the evidence side.

Maaret

--- In agile-testing@..., "Matthew" <matt.heusser@...> wrote:

>
>
> --- In agile-testing@..., "Maaret" <maaret@> wrote:
> >
> > This is a pretty strong statement with little evidence
> >provided for the community to judge for themselves
> >at least in my perspective.
> >
>
> Hey, Asking for evidence is reasonable.
>
> My blog post ( http://www.stpcollaboarative.com/matt ) has a link to the googlecached course outline as presented by the WAQB, which is a direct cut/paste of the outline from Lisa and Janet's book. It is provided without Attribution or permission; that is a copyright violation.
>
> Check lisa's twitter posts ( http://twitter.com/lisacrispin ); this is the /second/ time they've done this. In other words, they cut/pasted the outline before, were asked to put it down, waited six months, and put it back up again.
>
> Well, it's almost a cut/paste. Most likely, it's a re-type, as there are typographical errors.
>
> Do your own research. Draw your own conclusions.
>
>
>
> regards,
>
>
> Matthew Heusser
> Blog: http://blogs.stpcollaborative.com/matt
>

Re: Please be advised

2009-12-09T09:22:14Z

--- In agile-testing@..., "Maaret" <maaret@...> wrote:
>
> This is a pretty strong statement with little evidence
>provided for the community to judge for themselves
>at least in my perspective.
>

Hey, Asking for evidence is reasonable.

My blog post ( http://www.stpcollaboarative.com/matt ) has a link to the googlecached course outline as presented by the WAQB, which is a direct cut/paste of the outline from Lisa and Janet's book. It is provided without Attribution or permission; that is a copyright violation.

Check lisa's twitter posts ( http://twitter.com/lisacrispin ); this is the /second/ time they've done this. In other words, they cut/pasted the outline before, were asked to put it down, waited six months, and put it back up again.

Well, it's almost a cut/paste. Most likely, it's a re-type, as there are typographical errors.

Do your own research. Draw your own conclusions.

regards,

Matthew Heusser
Blog: http://blogs.stpcollaborative.com/matt

Re: Please be advised

2009-12-09T09:03:56Z

This is a pretty strong statement with little evidence provided for the community to judge for themselves at least in my perspective. I went to look for the materials, did not run across them online. There was most likely more behind this than just the course outline and there's been access to the course materials as well? I trust Lisa as an expert in this area, but would like to understand on a bit deeper level.

I enjoyed the book by Lisa and Janet a lot. Having been around with agile testing for a while, I did not see that all of it was new, but the way it was put together was clear and instructive.

Maaret

--- In agile-testing@..., Lisa Crispin <lisa.crispin@...> wrote:

>
> A company called Bouvet, alternatively called the "World Agile
> Qualifications Board", is marketing an agile testing course that is a direct
> rip-off of the book written by me and Janet, as well as our own course.
>
> If you are interested in agile testing training, there are plenty of
> legitimate sources of this, including our course, Elisabeth Hendrickson,
> Dale Emery, and I believe, ethical companies such as SQE and Rally, just to
> name a few. Don't give any business to people who don't actually know
> anything, but simply steal material from practitioners.
> Thanks,
> Lisa
>
> --
> Lisa Crispin
> Co-author with Janet Gregory, _Agile Testing: A Practical Guide for Testers
> and Agile Teams_ (Addison-Wesley 2009)
> http://lisacrispin.com
>

RE: Re: Please be advised

2009-12-09T06:50:57Z

Lisa and Janet - I wanted to let you know that I just purchased your book from Amazon. I've heard very good things about you work!

Melody Norsworthy

To: agile-testing@...
From: matt.heusser@...
Date: Wed, 9 Dec 2009 14:06:52 +0000
Subject: [agile-testing] Re: Please be advised

--- In agile-testing@..., Lisa Crispin <lisa.crispin@...> wrote:
>
> A company called Bouvet, alternatively called the "World Agile
> Qualifications Board", is marketing an agile testing course that is a direct rip-off of the book written by me and Janet,
> as well as our own course ...

I'm appealing directly to the Agile Alliance's Board of directors to formally censure this company. Proposal here:

http://blogs.stpcollaborative.com/matt/2009/12/08/nor-tolerate-those-among-us-who-do/

Let us not tolerate these fools and charlatans. I ask you to pllease consider signing the proposal, re-tweeting, linking, etc.

regards,

Matthew Heusser
Blog: http://blogs.stpocllaborative.com/matt

_________________________________________________________________
Windows Live Hotmail gives you a free,exclusive gift.
http://www.microsoft.com/windows/windowslive/hotmail_bl1/hotmail_bl1.aspx?ocid=PID23879::T:WLMTAGL:ON:WL:en-ww:WM_IMHM_7:092009

Re: Re: Please be advised

2009-12-09T06:18:58Z

As I mentioned in another message, Bouvet was contracting Steen
Lerche-Jensen of the WAQB to put on the Agile Testing Foundations
course. As far as I can tell, and from what they have told me, they
have no other affiliation with him. People from Bouvet have been in
contact with Lisa about the issue and are taking action.

As for the WAQB, Laurent Bossavit has said that there's no record of
them being an Agile Alliance member.

--

Dave Rooney
Co-founder and Consultant, The Agile Consortium
"Maximizing the value of your IT investments!"
E-mail: dave@...
Twitter: daverooneyca
http://www.theagileconsortium.com
http://practicalagility.blogspot.com

Matthew wrote:

> --- In agile-testing@..., Lisa Crispin <lisa.crispin@...> wrote:
>
>> A company called Bouvet, alternatively called the "World Agile
>> Qualifications Board", is marketing an agile testing course that is a direct rip-off of the book written by me and Janet,
>> as well as our own course ...
>>
>
> I'm appealing directly to the Agile Alliance's Board of directors to formally censure this company. Proposal here:
>
> http://blogs.stpcollaborative.com/matt/2009/12/08/nor-tolerate-those-among-us-who-do/
>
> Let us not tolerate these fools and charlatans. I ask you to pllease consider signing the proposal, re-tweeting, linking, etc.
>
>
> regards,
>
>
>
> Matthew Heusser
> Blog: http://blogs.stpocllaborative.com/matt
>

Re: Please be advised

2009-12-09T06:16:14Z

Just to be clear, Bouvet != WAQB. Bouvet was having Steen Lerche-Jensen
of WAQB put on the Agile Testing course.

Dave Rooney
Co-founder and Consultant, The Agile Consortium
"Maximizing the value of your IT investments!"
E-mail: dave@...
Twitter: daverooneyca
http://www.theagileconsortium.com
http://practicalagility.blogspot.com

Lisa Crispin wrote:

>
>
> A company called Bouvet, alternatively called the "World Agile
> Qualifications Board", is marketing an agile testing course that is a
> direct rip-off of the book written by me and Janet, as well as our own
> course.
>
> If you are interested in agile testing training, there are plenty of
> legitimate sources of this, including our course, Elisabeth
> Hendrickson, Dale Emery, and I believe, ethical companies such as SQE
> and Rally, just to name a few. Don't give any business to people who
> don't actually know anything, but simply steal material from
> practitioners.
> Thanks,
> Lisa
>
> --
> Lisa Crispin
> Co-author with Janet Gregory, _Agile Testing: A Practical Guide for
> Testers and Agile Teams_ (Addison-Wesley 2009)
> http://lisacrispin.com
>

Re: Can we always pre-prepare automated regression tests?

2009-12-09T06:10:56Z

--- In agile-testing@..., Steven Gordon <sgordonphd@...> wrote
>The question is whether it is reasonable to expect to be
>able to go from high level acceptance criteria to first
>cut unambiguous criteria to additional unambiguous criteria
>discovered by exploratory testing (not to mention some
>direct feedback from the Product Manager) all under the
>umbrella of the same single story and get it completely
>done in a single iteration (along with several other stories).
>

We've been doing it at Socialtext for 37 out of the past 41 two-week iterations (with a couple long iterations around the holidays.)

But we expect regression testing to spill over into the next iteration, and estimation and planning on the next iteration start early.

Clearly, there's more than one way to do it.

regards,

--heusser

Re: Please be advised

2009-12-09T06:06:52Z

--- In agile-testing@..., Lisa Crispin <lisa.crispin@...> wrote:
>
> A company called Bouvet, alternatively called the "World Agile
> Qualifications Board", is marketing an agile testing course that is a direct rip-off of the book written by me and Janet,
> as well as our own course ...

I'm appealing directly to the Agile Alliance's Board of directors to formally censure this company. Proposal here:

http://blogs.stpcollaborative.com/matt/2009/12/08/nor-tolerate-those-among-us-who-do/

Let us not tolerate these fools and charlatans. I ask you to pllease consider signing the proposal, re-tweeting, linking, etc.

regards,

Matthew Heusser
Blog: http://blogs.stpocllaborative.com/matt

Call for Papers - Test Driven Development 2010 - colocated with ICST 2010

2009-12-08T18:51:20Z

---------- Forwarded message ----------
From: Laurie Williams <williams@...>
Date: Mon, Dec 7, 2009 at 1:34 PM
Subject: [SEWORLD] Call for Papers - Test Driven Development 2010 -
colocated with ICST 2010
To: seworld@...

> Test Driven Development 2010
> http://agile.csc.ncsu.edu/tdd/
>
> Paris, France
> April 10, 2010
>
> Co-located with ICST 2010
> http://vps.it-sudparis.eu/icst2010/icst2010v4.htm
>
> Test-driven development (TDD) emerged as a practice of Extreme Programming

approximately a decade ago. The TDD practice involves the incremental
automation of unit test cases and the crafting of often-automated acceptance
tests early in an iteration cycle. The goal of the TDD 2010 workshop is to
bring together academics researching TDD, practitioners using TDD, educators
teaching TDD, and researchers interested in TDD, to discuss new and emerging
trends in TDD and to assess the advancement of TDD as a practice for
improving the quality of software products. We invite submissions of both
full-length and short papers as well as proposals for presentations. The
workshop will include these presentations and interactive discussion time.
>
> **TDD 2010 topics of interest:
> * Experience of industrial teams using test-driven development
> * Empirical studies of industrial teams using test-driven development
> * Tools to support test-driven development, particularly those beyond the
initial tool, JUnit
> * Guidelines on how much test-driven development test cases are enough
> * The effects of test-driven development on the design of software
> * Use of test-driven development test cases as executable documentation
> * Experience starting test-driven development on legacy code bases
without automated tests
> * Education in TDD practices
>
> **Submissions:
> TDD 2010 will have three categories of submissions:
> · Long papers (up to 10 pages)
> · Short papers (up to 6 pages)
> · Presentation proposals (one page; not in Digital Library)
> Each submitted paper must adhere to the IEEE template. Three members of
the program committee will review both short and long papers; presentation
proposals will be evaluated by the chairs for relevance to the workshop

>
> ** Important dates:
> - Submission of papers: January 22, 2010
> - Notification: March 2, 2010
> - Camera ready: March 26, 2010
>
> ** Organizing Committee:
> Laurie Williams, North Carolina State University, USA
> Jeff Offutt, George Mason University, USA
>
> ** Program committee:
> Jan Acosta, IBM
> Lisa Crispen, lisacrispen.com
> David Jenzen, Cal Poly
> Philip Johnson, University of Hawaii
> Catherine Louis
> Michael Maximilien, IBM
> Grigori Melnick, Microsoft
> Gerard Meszaros, gerardmeszaros,com

******************************************

Laurie Williams, PhD
North Carolina State University
Department of Computer Science
Campus Box 8206
890 Oval Drive, Room 3272
Raleigh, NC 27695

Phone: 919-513-4151
Fax: 919-515-7896

============================================================
To contribute to SEWORLD, send your submission to
mailto:seworld@...

http://www.sigsoft.org/seworld provides more
information on SEWORLD as well as a complete archive of
messages posted to the list.
============================================================

Re: Re: Can we always pre-prepare automated regression tests?

2009-12-08T17:40:38Z

Hi Steve,

The question is whether it is reasonable to expect to be able to go from
> high level acceptance criteria to first cut unambiguous criteria to
> additional unambiguous criteria discovered by exploratory testing (not to
> mention some direct feedback from the Product Manager) all under the
> umbrella of the same single story and get it completely done in a single
> iteration (along with several other stories).
>

I can't speak for others in the conversation, but I wasn't thinking that
this all had to be done in a single iteration.

> Attempting this creates greater stress, a more adversarial relationship
> between testers and developers, and a less sustainable process compared
> with:
> - slicing the original story along the lines of the specific test cases,
> - scheduling them individually within the iteration or next few iterations,
>

Yes, my preference write the test cases and slice the stories just before
the iteration in which they will be implemented.

Something I don't see in your description: Who is involved in writing the
test cases?

I want to involve not only the Product Owner (or whoever's job it is to
decide what they want the software to do), but also testers and developers.
Testers are good at identifying ambiguities in the story description, and
positing scenarios to test everyone's understanding of the story. Developers
are good at understanding the technical implications of a story or a test,
and at understanding how the existing technology influences the cost of the
next story. And that can influence what tests the team needs to write.

So what comes out of that conversation is not only the unambiguous test
cases, but also a shared understanding of what the story means, and what the
tests mean.

- using exploratory testing to define missing slice stories,
>

I'm not clear what you mean here. Is the work of defining missing slice
stories the same or different from the work you mentioned above, discovering
additional unambiguous criteria? If they're different, how are they
different? If they're the same, how do you avoid the chaos you mentioned
above?

Dale

--
Dale Emery
Consultant to software teams and leaders
Web: http://dhemery.com
Weblog: http://cwd.dhemery.com

Re: Re: Can we always pre-prepare automated regression tests?

2009-12-08T16:01:43Z

Of course, the team should absolutely be able to advance the product this
way.

The question is whether it is reasonable to expect to be able to go from
high level acceptance criteria to first cut unambiguous criteria to
additional unambiguous criteria discovered by exploratory testing (not to
mention some direct feedback from the Product Manager) all under the
umbrella of the same single story and get it completely done in a single
iteration (along with several other stories).

Attempting this creates greater stress, a more adversarial relationship
between testers and developers, and a less sustainable process compared
with:
- slicing the original story along the lines of the specific test cases,
- scheduling them individually within the iteration or next few iterations,
- using exploratory testing to define missing slice stories,
- continually getting feedback as to whether the slice stories are providing
value.

It is the same work as you describe, but less chaotic and more sustainable.

Once you have a slice story for each acceptance test, then the acceptance
test becomes the requirement for that story. You can still have the
original ambiguous requirement for the epic, but there is no reason to
maintain that requirement once everyone agrees that all the slices necessary
to fulfil the original epic requirement have been implemented successfully.
Then the acceptance tests are all you need to make sure the epic requirement
stays satisfied.

On Tue, Dec 8, 2009 at 4:00 PM, adam_peter.knight <adam.knight@...
> wrote:

>
>
> I love how this discussion has developed! My two-penneth:-
>
> When tackling a story our method tends to be to define a set of high level
> test acceptance criteria and agree these with the Product Manager and
> developers prior to defining specific test cases. It is at this stage that I
> would expect ambiguities or misunderstandings in the requirement to be
> flushed out, e.g. the absence of the rule defining behaviour as in Dale's
> previous example. A good tester should identify edge cases and ambiguities
> at this stage and clarify these with more precise language in the acceptance
> criteria. Specific tests would then be defined to give us confidence that
> the criteria were being met, but I would not expect the passing of the tests
> to be the requirement, merely the mechanism that we have chosen to
> demonstrate that the requirement is being met. We might (should), based on
> the acceptance criteria, be able to execute further exploratory tests not
> previously agreed, and use our acceptance criteria to decide whether these
> tests have been successful or not.
>
> Adam.
>
> --------------------------------
> http://www.linkedin.com/in/adampknight
> http://a-sisyphean-task.blogspot.com
> http://www.twitter.com/adampknight
> --------------------------------
>
> "The true test of character is not how much we know how to do, but how we
> behave when we don't know what to do." John W. Holt, Jr.
>
>
> --- In agile-testing@..., Dale Emery <dale@...> wrote:
> >
> > Hi George,
> >
> > As an aside, this test is erroneous only because you have decided it so.
> It
> > > /could have been/ that the test was correct, but that the requirement
> was
> > > stated incorrectly. People are often not precise in their statements,
> as in
> > > their interpretation of statements.
> > >
> >
> > I think I agree, though I'd say it slightly differently: It's erroneous
> > because (in this particular case) it did not express my true intent.
> >
> > I definitely agree that a mismatch per se doesn't necessarily mean that
> the
> > test is erroneous. The error might be in the test, the requirements
> > statement, or both.
> >
> > My point in pointing this out is to highlight the fact that, though the
> > > examples are not, in themselves, the statement of requirements,
> well-chosen
> > > examples can help us avoid misunderstandings of those requirements.
> > >
> >
> > Yes. I'd say that the two expressions complement each other. Each
> expression
> > helps the reader to understand the other, and to understand the
> underlying
> > intent, which is incompletely expressed by either form.
> >
> > Dale
> >
> > --
> > Dale Emery
> > Consultant to software teams and leaders
> > Web: http://dhemery.com
> > Weblog: http://cwd.dhemery.com
> >
>
>
>

Re: Can we always pre-prepare automated regression tests?

2009-12-08T15:00:23Z

I love how this discussion has developed! My two-penneth:-

When tackling a story our method tends to be to define a set of high
level test acceptance criteria and agree these with the Product Manager
and developers prior to defining specific test cases. It is at this
stage that I would expect ambiguities or misunderstandings in the
requirement to be flushed out, e.g. the absence of the rule defining
behaviour as in Dale's previous example. A good tester should identify
edge cases and ambiguities at this stage and clarify these with more
precise language in the acceptance criteria. Specific tests would then
be defined to give us confidence that the criteria were being met, but I
would not expect the passing of the tests to be the requirement, merely
the mechanism that we have chosen to demonstrate that the requirement is
being met. We might (should), based on the acceptance criteria, be able
to execute further exploratory tests not previously agreed, and use our
acceptance criteria to decide whether these tests have been successful
or not.

Adam.

--------------------------------
http://www.linkedin.com/in/adampknight
http://a-sisyphean-task.blogspot.com
http://www.twitter.com/adampknight
--------------------------------

"The true test of character is not how much we know how to do, but how
we behave when we don't know what to do." John W. Holt, Jr.

--- In agile-testing@..., Dale Emery <dale@...> wrote:
>
> Hi George,
>
> As an aside, this test is erroneous only because you have decided it
so. It
> > /could have been/ that the test was correct, but that the
requirement was
> > stated incorrectly. People are often not precise in their
statements, as in
> > their interpretation of statements.
> >
>
> I think I agree, though I'd say it slightly differently: It's
erroneous
> because (in this particular case) it did not express my true intent.
>
> I definitely agree that a mismatch per se doesn't necessarily mean
that the
> test is erroneous. The error might be in the test, the requirements
> statement, or both.
>
> My point in pointing this out is to highlight the fact that, though
the
> > examples are not, in themselves, the statement of requirements,
well-chosen
> > examples can help us avoid misunderstandings of those requirements.
> >
>
> Yes. I'd say that the two expressions complement each other. Each
expression
> helps the reader to understand the other, and to understand the
underlying

> intent, which is incompletely expressed by either form.
>
> Dale
>
> --
> Dale Emery
> Consultant to software teams and leaders
> Web: http://dhemery.com
> Weblog: http://cwd.dhemery.com
>

Re: Please be advised

2009-12-08T11:00:42Z

If you haven't picked it up yet, Lisa & Janet's book is outstanding. Please
do not support people who didn't do the work.

Thanks again for a reference that I use weekly.

-Annette

On Tue, Dec 8, 2009 at 10:20 AM, Vesna Leonard <vesna.leonard@...>wrote:

>
>
> Hi Lisa – I’ve known him for a few years now and we’re trying to hook up
> to work together. It will happen though! ‘Till then, I must live
> vicariously through Selena Delesie, who is currently working with him….
>
>
>
> Will do re the agile journey!
>
>
>
> Ves
>
>
>
> *Vesna** Leonard*
>
> *SQA** Professional*
>
> * *
>
> *[image: LogoSmall]*
>
> *Personalized **Corporate Services *
>
> * *
>
> Cell: *226-339-6045*
>
> Email: *vesna@...*
>
> Website:* www.worx4you.com*
>
> LinkedIn: *www.linkedin.com/in/vesnaworx4you*<http://www.linkedin.com/in/vesnaworx4you>
>
> Twitter: *www.twitter.com/worx4you*<http://www.linkedin.com/in/vesnaworx4you>
>
>
>
> *From:* agile-testing@... [mailto:
> agile-testing@...] *On Behalf Of *Lisa Crispin
> *Sent:* Tuesday, December 08, 2009 1:13 PM
>
> *To:* agile-testing@...
> *Subject:* Re: [agile-testing] Please be advised
>
>
>
>
>
> Hi Ves,
> Oh, Declan is a wonderful agile coach, that is too bad! Please keep us
> posted on your journey to agile!
> -- Lisa
>
> On Tue, Dec 8, 2009 at 10:51 AM, Vesna Leonard <vesna.leonard@...>
> wrote:
>
>
>
> Yikes! Thanks for the info Lisa. I just ordered your book a few days ago
> as I’m finally free of hard-core waterfall companies J I am starting to
> work with a client that actively wants to move from waterfall to Agile
> (bringing in an Agile coach as well – was supposed to be Declan Whelan, but
> unfortunately the timing isn’t right there…). Anyway, very exciting and I
> can’t wait for your book to come in. Good to know which sources to stay
> away from in unfamiliar territory J
>
>
>
> Ves
>
>
>
>
>
> *Vesna** Leonard*
>
> *SQA** Professional*
>
> * *
>
> *[image: LogoSmall]*
>
> *Personalized **Corporate Services *
>
> * *
>
> Cell: *226-339-6045*
>
> Email: *vesna@...*
>
> Website:* www.worx4you.com*
>
> LinkedIn: *www.linkedin.com/in/vesnaworx4you*<http://www.linkedin.com/in/vesnaworx4you>
>
> Twitter: *www.twitter.com/worx4you*<http://www.linkedin.com/in/vesnaworx4you>
>
>
>
> *From:* agile-testing@... [mailto:
> agile-testing@...] *On Behalf Of *Lisa Crispin
> *Sent:* Tuesday, December 08, 2009 12:37 PM
> *To:* agile-testing@...
> *Subject:* [agile-testing] Please be advised
>
>
>
>
>
> A company called Bouvet, alternatively called the "World Agile
> Qualifications Board", is marketing an agile testing course that is a direct
> rip-off of the book written by me and Janet, as well as our own course.
>
> If you are interested in agile testing training, there are plenty of
> legitimate sources of this, including our course, Elisabeth Hendrickson,
> Dale Emery, and I believe, ethical companies such as SQE and Rally, just to
> name a few. Don't give any business to people who don't actually know
> anything, but simply steal material from practitioners.
> Thanks,
> Lisa
>
> --
> Lisa Crispin
> Co-author with Janet Gregory, _Agile Testing: A Practical Guide for Testers
> and Agile Teams_ (Addison-Wesley 2009)
> http://lisacrispin.com
>
>
>
>
> --
> Lisa Crispin
> Co-author with Janet Gregory, _Agile Testing: A Practical Guide for Testers
> and Agile Teams_ (Addison-Wesley 2009)
> http://lisacrispin.com
>
>
>

image001.png (11K) Download Attachment

RE: Please be advised

2009-12-08T10:20:47Z

Hi Lisa - I've known him for a few years now and we're trying to hook up to
work together. It will happen though! 'Till then, I must live vicariously
through Selena Delesie, who is currently working with him..

Will do re the agile journey!

Ves

Vesna Leonard

SQA Professional

LogoSmall

Personalized Corporate Services

Cell: 226-339-6045

Email: <mailto:vesna@...> vesna@...

Website: <http://www.worx4you.com> www.worx4you.com

LinkedIn: <http://www.linkedin.com/in/vesnaworx4you>
www.linkedin.com/in/vesnaworx4you

Twitter: <http://www.linkedin.com/in/vesnaworx4you>
www.twitter.com/worx4you

From: agile-testing@... [mailto:agile-testing@...]
On Behalf Of Lisa Crispin
Sent: Tuesday, December 08, 2009 1:13 PM
To: agile-testing@...
Subject: Re: [agile-testing] Please be advised

Hi Ves,
Oh, Declan is a wonderful agile coach, that is too bad! Please keep us
posted on your journey to agile!
-- Lisa

On Tue, Dec 8, 2009 at 10:51 AM, Vesna Leonard <vesna.leonard@...>
wrote:

Yikes! Thanks for the info Lisa. I just ordered your book a few days ago
as I'm finally free of hard-core waterfall companies J I am starting to
work with a client that actively wants to move from waterfall to Agile
(bringing in an Agile coach as well - was supposed to be Declan Whelan, but
unfortunately the timing isn't right there.). Anyway, very exciting and I
can't wait for your book to come in. Good to know which sources to stay
away from in unfamiliar territory J

Ves

Vesna Leonard

SQA Professional

LogoSmall

Personalized Corporate Services

Cell: 226-339-6045

Email: vesna@...

Website: www.worx4you.com

LinkedIn: <http://www.linkedin.com/in/vesnaworx4you>
www.linkedin.com/in/vesnaworx4you

Twitter: <http://www.linkedin.com/in/vesnaworx4you>
www.twitter.com/worx4you

From: agile-testing@... [mailto:agile-testing@...]
On Behalf Of Lisa Crispin
Sent: Tuesday, December 08, 2009 12:37 PM
To: agile-testing@...
Subject: [agile-testing] Please be advised

A company called Bouvet, alternatively called the "World Agile
Qualifications Board", is marketing an agile testing course that is a direct
rip-off of the book written by me and Janet, as well as our own course.

If you are interested in agile testing training, there are plenty of
legitimate sources of this, including our course, Elisabeth Hendrickson,
Dale Emery, and I believe, ethical companies such as SQE and Rally, just to
name a few. Don't give any business to people who don't actually know
anything, but simply steal material from practitioners.
Thanks,
Lisa

--
Lisa Crispin
Co-author with Janet Gregory, _Agile Testing: A Practical Guide for Testers
and Agile Teams_ (Addison-Wesley 2009)
http://lisacrispin.com

--
Lisa Crispin
Co-author with Janet Gregory, _Agile Testing: A Practical Guide for Testers
and Agile Teams_ (Addison-Wesley 2009)
http://lisacrispin.com

image001.png (11K) Download Attachment

Re: Please be advised

2009-12-08T10:12:54Z

Hi Ves,
Oh, Declan is a wonderful agile coach, that is too bad! Please keep us
posted on your journey to agile!
-- Lisa

On Tue, Dec 8, 2009 at 10:51 AM, Vesna Leonard <vesna.leonard@...>wrote:

>
>
> Yikes! Thanks for the info Lisa. I just ordered your book a few days
> ago as I’m finally free of hard-core waterfall companies J I am starting
> to work with a client that actively wants to move from waterfall to Agile
> (bringing in an Agile coach as well – was supposed to be Declan Whelan, but
> unfortunately the timing isn’t right there…). Anyway, very exciting and I
> can’t wait for your book to come in. Good to know which sources to stay
> away from in unfamiliar territory J
>
>
>
> Ves
>
>
>
>
>
> *Vesna** Leonard*
>
> *SQA** Professional*
>
> * *
>
> *[image: LogoSmall]*
>
> *Personalized **Corporate Services *
>
> * *
>
> Cell: *226-339-6045*
>
> Email: *vesna@...*
>
> Website:* www.worx4you.com*
>
> LinkedIn: *www.linkedin.com/in/vesnaworx4you*<http://www.linkedin.com/in/vesnaworx4you>
>
> Twitter: *www.twitter.com/worx4you*<http://www.linkedin.com/in/vesnaworx4you>
>
>
>
> *From:* agile-testing@... [mailto:
> agile-testing@...] *On Behalf Of *Lisa Crispin
> *Sent:* Tuesday, December 08, 2009 12:37 PM
> *To:* agile-testing@...
> *Subject:* [agile-testing] Please be advised
>
>
>
>
>
> A company called Bouvet, alternatively called the "World Agile
> Qualifications Board", is marketing an agile testing course that is a direct
> rip-off of the book written by me and Janet, as well as our own course.
>
> If you are interested in agile testing training, there are plenty of
> legitimate sources of this, including our course, Elisabeth Hendrickson,
> Dale Emery, and I believe, ethical companies such as SQE and Rally, just to
> name a few. Don't give any business to people who don't actually know
> anything, but simply steal material from practitioners.
> Thanks,
> Lisa
>
> --
> Lisa Crispin
> Co-author with Janet Gregory, _Agile Testing: A Practical Guide for Testers
> and Agile Teams_ (Addison-Wesley 2009)
> http://lisacrispin.com
>
>
>

--
Lisa Crispin
Co-author with Janet Gregory, _Agile Testing: A Practical Guide for Testers
and Agile Teams_ (Addison-Wesley 2009)
http://lisacrispin.com

image001.png (11K) Download Attachment

RE: Please be advised

2009-12-08T09:51:20Z

Yikes! Thanks for the info Lisa. I just ordered your book a few days ago
as I'm finally free of hard-core waterfall companies J I am starting to
work with a client that actively wants to move from waterfall to Agile
(bringing in an Agile coach as well - was supposed to be Declan Whelan, but
unfortunately the timing isn't right there.). Anyway, very exciting and I
can't wait for your book to come in. Good to know which sources to stay
away from in unfamiliar territory J

Ves

Vesna Leonard

SQA Professional

LogoSmall

Personalized Corporate Services

Cell: 226-339-6045

Email: <mailto:vesna@...> vesna@...

Website: <http://www.worx4you.com> www.worx4you.com

LinkedIn: <http://www.linkedin.com/in/vesnaworx4you>
www.linkedin.com/in/vesnaworx4you

Twitter: <http://www.linkedin.com/in/vesnaworx4you>
www.twitter.com/worx4you

From: agile-testing@... [mailto:agile-testing@...]
On Behalf Of Lisa Crispin
Sent: Tuesday, December 08, 2009 12:37 PM
To: agile-testing@...
Subject: [agile-testing] Please be advised

A company called Bouvet, alternatively called the "World Agile
Qualifications Board", is marketing an agile testing course that is a direct
rip-off of the book written by me and Janet, as well as our own course.

If you are interested in agile testing training, there are plenty of
legitimate sources of this, including our course, Elisabeth Hendrickson,
Dale Emery, and I believe, ethical companies such as SQE and Rally, just to
name a few. Don't give any business to people who don't actually know
anything, but simply steal material from practitioners.
Thanks,
Lisa

--
Lisa Crispin
Co-author with Janet Gregory, _Agile Testing: A Practical Guide for Testers
and Agile Teams_ (Addison-Wesley 2009)
http://lisacrispin.com

image001.png (11K) Download Attachment

Please be advised

2009-12-08T09:37:07Z

A company called Bouvet, alternatively called the "World Agile
Qualifications Board", is marketing an agile testing course that is a direct
rip-off of the book written by me and Janet, as well as our own course.

If you are interested in agile testing training, there are plenty of
legitimate sources of this, including our course, Elisabeth Hendrickson,
Dale Emery, and I believe, ethical companies such as SQE and Rally, just to
name a few. Don't give any business to people who don't actually know
anything, but simply steal material from practitioners.
Thanks,
Lisa

--
Lisa Crispin
Co-author with Janet Gregory, _Agile Testing: A Practical Guide for Testers
and Agile Teams_ (Addison-Wesley 2009)
http://lisacrispin.com

Re: Can we always pre-prepare automated regression tests?

2009-12-07T07:55:28Z

+1

I'll add this. At the end of the day, if your IT is like most IT shops, the "goodness" of the software is defined by the tests. If the software passes all tests, then it can be loaded into production.

In some cases the failing test(s) simply point out that some new requirement has superseded an old requirement, so the business has to decide which rule now applies. The tests are updated accordingly.

If the software fails in production, then we have to look to see what we missed. Typically there's a scenario or two that was overlooked. OK, so we add a few more tests to the test suite. Lather, rinse, repeat.

We can talk about intent all day, but when you're dealing with complex systems, and thousands of requirements, the automated tests are the gatekeepers. The tests may not be the "real" requirements, but the 20,000 test cases in our test suite are the best proxy we have.

Mark

--- In agile-testing@..., Steven Gordon <sgordonphd@...> wrote:

>
> Understanding intent is important. It is so important, we should not trust
> it to documents. We should only trust direct human interaction between
> customers, testers and developers to establish that understanding, and then
> trust feedback to working software to verify that the intent was satisfied.
>
> Documenting intent in enough detail for it to be a "requirement" is a waste
> of time and effort. What needs to be documented in detail is what
> unambiguous targets come out of the discussion between customers, testers
> and developers. These targets serve to define and focus the development
> effort between the conversation and the delivery of working software. These
> targets are the "requirements" during the iteration. The intent is just
> very useful background information.
>
> Feedback to the working software (from customers and/or exploratory testing)
> that indicate intent was not fully met should result in another round of
> conversation, unambiguous targets, delivery and feedback.
>
> It is not a waste to "document" these unambiguous targets that delivered
> software is being developed to meet. They serve as regression tests going
> forward.
>
> If we need human readable documentation of what the software does, I agree
> that annotations on tests are good but insufficient. I have found that the
> user's guide works much better than the historic trail of intent documents.
> Keeping the user's guide up to date is not waste, because we have to keep it
> up to date for it to be useful to users anyway.
>
> ===============================================================================
>
> While understanding the intent is important, ultimately what matters is
> whether the software does what it is supposed to do. It is quite
> conceivable that the software can do what it is supposed to do while the
> intent that was conveyed was really a 'fairy tale' because the domain was so
> complex it would take years of course work to understand the real intent.
>
> We do not need to understand the secret formula behind the dice
> interpretation to implement correctly working software as long as we have
> enough examples to hone in on something that works reliably. Even if we are
> given the secret formula we still might not understand the real intent.
>
> For example, suppose the dice interpretations actually describe a certain
> behavior of quarks. Do we really need to study Physics for enough years to
> understand the theory of Quarks to be able to write and test this software?
>
> Much of my work has been in rule-based systems. I have had many customers
> who would throw rules over the wall and say just implement those rules. I
> encounter resistance when I say I can implement those rules and test whether
> those rules do what they are supposed to do in isolation, but I cannot
> assure that in the context of thousands of other rules that these rules will
> really have the effect they are looking for without sufficient concrete
> examples of how the results should be different for the overall software.
> In other words, I can be meeting their intent, but I cannot be sure it is
> working right without the examples.
>
> ===================================================================================
>
> So, which are the requirements, the intents or the examples?
>
> My 30 years of experience developing software in academic, industrial and
> commercial contexts lead me to very much believe that considering the
> intents the requirements and the examples to just be tests leads to software
> development processes that are more chaotic and adversarial than necessary.
> Just considering the examples to be the requirements and the intents to be
> valuable background information makes the exact same development work done
> in the exact same order by the exact same people significantly less chaotic
> and adversarial.
>
> Steven Gordon
>
> On Sat, Dec 5, 2009 at 3:37 PM, Dale Emery <dale@...> wrote:
>
> >
> >
> > Hi George,
> >
> > As an aside, this test is erroneous only because you have decided it so.
> >> It /could have been/ that the test was correct, but that the requirement was
> >> stated incorrectly. People are often not precise in their statements, as in
> >> their interpretation of statements.
> >>
> >
> > I think I agree, though I'd say it slightly differently: It's erroneous
> > because (in this particular case) it did not express my true intent.
> >
> > I definitely agree that a mismatch per se doesn't necessarily mean that the
> > test is erroneous. The error might be in the test, the requirements
> > statement, or both.
> >
> > My point in pointing this out is to highlight the fact that, though the
> >> examples are not, in themselves, the statement of requirements, well-chosen
> >> examples can help us avoid misunderstandings of those requirements.
> >>
> >
> > Yes. I'd say that the two expressions complement each other. Each
> > expression helps the reader to understand the other, and to understand the
> > underlying intent, which is incompletely expressed by either form.
> >
> > Dale
> >
> > --
> > Dale Emery
> > Consultant to software teams and leaders
> > Web: http://dhemery.com
> > Weblog: http://cwd.dhemery.com
> >
> >
> >
>

Re: Can we always pre-prepare automated regression tests?

2009-12-07T03:01:37Z

>My opinion is that requirement documents are necessary,
>especially for larger organizations. Remember these documents
>are not only shared with QA and Development but are also used
>by technical writers, business analysts, account managers,
>marketing, sales, and PMs (project, program, and product
>managers). They provide an easy to understand way to provide this
> information to all teams.

At Socialtext, we hire a technical writer to produce external "help" documentation that satisfies this goal.

Because technical writers no how to write. The folks assembling the software are trying to make it work.

Your Mileage my vary; that's just me talkin'

--heusser

keynote from xpday 2009

2009-12-07T02:26:03Z

Mark Striebeck from Google did a very interesting keynote today at XpDay
2009, talking about what Google does to improve their testing practices.
Here's a write-up.

http://gojko.net/2009/12/07/improving-testing-practices-at-google/

--
gojko adzic
http://gojko.net

2nd edition of my fitnesse book now online

2009-12-06T17:19:27Z

Hi,

As of now, the second edition of Test Driven .NET Development with
FitNesse is free online. You can download the full PDF version or read
the book online in HTML at http://gojko.net/fitnesse.

You can read what's new and changed at
http://gojko.net/2009/12/07/fitnesse-book-now-free-online/

--
gojko adzic
http://gojko.net

Re: Can we always pre-prepare automated regression tests?

2009-12-06T05:49:32Z

Understanding intent is important. It is so important, we should not trust
it to documents. We should only trust direct human interaction between
customers, testers and developers to establish that understanding, and then
trust feedback to working software to verify that the intent was satisfied.

Documenting intent in enough detail for it to be a "requirement" is a waste
of time and effort. What needs to be documented in detail is what
unambiguous targets come out of the discussion between customers, testers
and developers. These targets serve to define and focus the development
effort between the conversation and the delivery of working software. These
targets are the "requirements" during the iteration. The intent is just
very useful background information.

Feedback to the working software (from customers and/or exploratory testing)
that indicate intent was not fully met should result in another round of
conversation, unambiguous targets, delivery and feedback.

It is not a waste to "document" these unambiguous targets that delivered
software is being developed to meet. They serve as regression tests going
forward.

If we need human readable documentation of what the software does, I agree
that annotations on tests are good but insufficient. I have found that the
user's guide works much better than the historic trail of intent documents.
Keeping the user's guide up to date is not waste, because we have to keep it
up to date for it to be useful to users anyway.

===============================================================================

While understanding the intent is important, ultimately what matters is
whether the software does what it is supposed to do. It is quite
conceivable that the software can do what it is supposed to do while the
intent that was conveyed was really a 'fairy tale' because the domain was so
complex it would take years of course work to understand the real intent.

We do not need to understand the secret formula behind the dice
interpretation to implement correctly working software as long as we have
enough examples to hone in on something that works reliably. Even if we are
given the secret formula we still might not understand the real intent.

For example, suppose the dice interpretations actually describe a certain
behavior of quarks. Do we really need to study Physics for enough years to
understand the theory of Quarks to be able to write and test this software?

Much of my work has been in rule-based systems. I have had many customers
who would throw rules over the wall and say just implement those rules. I
encounter resistance when I say I can implement those rules and test whether
those rules do what they are supposed to do in isolation, but I cannot
assure that in the context of thousands of other rules that these rules will
really have the effect they are looking for without sufficient concrete
examples of how the results should be different for the overall software.
In other words, I can be meeting their intent, but I cannot be sure it is
working right without the examples.

===================================================================================

So, which are the requirements, the intents or the examples?

My 30 years of experience developing software in academic, industrial and
commercial contexts lead me to very much believe that considering the
intents the requirements and the examples to just be tests leads to software
development processes that are more chaotic and adversarial than necessary.
Just considering the examples to be the requirements and the intents to be
valuable background information makes the exact same development work done
in the exact same order by the exact same people significantly less chaotic
and adversarial.

Steven Gordon

On Sat, Dec 5, 2009 at 3:37 PM, Dale Emery <dale@...> wrote:

>
>
> Hi George,
>
> As an aside, this test is erroneous only because you have decided it so.
>> It /could have been/ that the test was correct, but that the requirement was
>> stated incorrectly. People are often not precise in their statements, as in
>> their interpretation of statements.
>>
>
> I think I agree, though I'd say it slightly differently: It's erroneous
> because (in this particular case) it did not express my true intent.
>
> I definitely agree that a mismatch per se doesn't necessarily mean that the
> test is erroneous. The error might be in the test, the requirements
> statement, or both.
>
> My point in pointing this out is to highlight the fact that, though the
>> examples are not, in themselves, the statement of requirements, well-chosen
>> examples can help us avoid misunderstandings of those requirements.
>>
>
> Yes. I'd say that the two expressions complement each other. Each
> expression helps the reader to understand the other, and to understand the
> underlying intent, which is incompletely expressed by either form.
>
> Dale
>
> --
> Dale Emery
> Consultant to software teams and leaders
> Web: http://dhemery.com
> Weblog: http://cwd.dhemery.com
>
>
>

Re: Can we always pre-prepare automated regression tests?

2009-12-05T14:37:18Z

Hi George,

As an aside, this test is erroneous only because you have decided it so. It
> /could have been/ that the test was correct, but that the requirement was
> stated incorrectly. People are often not precise in their statements, as in
> their interpretation of statements.
>

I think I agree, though I'd say it slightly differently: It's erroneous
because (in this particular case) it did not express my true intent.

I definitely agree that a mismatch per se doesn't necessarily mean that the
test is erroneous. The error might be in the test, the requirements
statement, or both.

My point in pointing this out is to highlight the fact that, though the
> examples are not, in themselves, the statement of requirements, well-chosen
> examples can help us avoid misunderstandings of those requirements.
>

Yes. I'd say that the two expressions complement each other. Each expression
helps the reader to understand the other, and to understand the underlying
intent, which is incompletely expressed by either form.

Dale

--
Dale Emery
Consultant to software teams and leaders
Web: http://dhemery.com
Weblog: http://cwd.dhemery.com

Re: Can we always pre-prepare automated regression tests?

2009-12-05T14:18:17Z

Dale Emery wrote:
> Remember my example from earlier:
> Preferred customers receive a 10 percent discount on purchases over
> $1000 dollars. Many people, in writing tests to express this policy,
> would create a test that says: "If the customer is preferred, and the
> purchase amount is $1000 dollars, offer the discount." This test is
> erroneous.

As an aside, this test is erroneous only because you have decided it so.
It /could have been/ that the test was correct, but that the
requirement was stated incorrectly. People are often not precise in
their statements, as in their interpretation of statements.

My point in pointing this out is to highlight the fact that, though the
examples are not, in themselves, the statement of requirements,
well-chosen examples can help us avoid misunderstandings of those
requirements.

- George

--
----------------------------------------------------------------------
* George Dinwiddie * http://blog.gdinwiddie.com
Software Development http://www.idiacomputing.com
Consultant and Coach http://www.agilemaryland.org
----------------------------------------------------------------------

Re: Can we always pre-prepare automated regression tests?

2009-12-05T13:25:22Z

Hi Steve,

Ultimately, if requirements have to be testable, then why not consider the
> tests themselves to be the requirements and avoid the overhead and confusion
> of having requirements in other ambiguous, incomplete forms.
>

My reasoning: Tests may not clearly indicate the intent, and tests are an
incomplete expression of intent.

A few months ago, I watched as George Dinwiddie demonstrated a JamesBachian
dice-game exercise with a colleague, Paul. The game went like this:

1. Paul rolls some dice, or arranges them deliberately, and says what number
he thinks George will announce based on the dice.

2. Based on the dice, George applies a consistent rule (to which Paul is not
privy) to derive a single number, and announces the number to Paul.

3. Repeat until Paul can state the rule that George is using to derive the
number.

4. John and Ringo were elsewhere at the time.

After a dozen or so rounds, Paul (a very bright guy) was unable to state the
rule. But he had constructed a table, with which he could accurately and
reliably predict what number George would announce. George was apparently
summing some numbers, but not the face values on the dice -- instead he was
summing some other numbers that related to the values on the dice.

Paul's table looked like this (the number on the left is the die value, the
number on the right is the number that George included in the sum):

1 -> 0
2 -> 0
3 -> 2
4 -> 0
5 -> 4
6 -> 0

So if Paul rolled 1 4 5 3, he could reliably predict that George would say
6. But he did not know why. He didn't know what rule George was applying.

I had an aha as I watched: This is analogous to tests and requirements. Paul
was able to pass every test with those dice. But he was not able to derive
the rule after running scores of tests.

Now, I happened to know the rule that George was using. Not because he told
me, but because I had seen others use the rule before.

Another aspect of this is that no set of tests could be complete. Sure,
there is a limited number of configurations of four regular dice, and you
could try every one of those. But suppose George now added dice with values
larger than 6? Suppose George introduced dice with seven, eight, or nine
pips on some sides. How should those values sum into the total? If you know
the rule, you can predict the result. If you don't know the rule, you can't.

Also, if we don't know the rule that generates the tests, how can we tell
whether the tests are correct? Remember my example from earlier: Preferred
customers receive a 10 percent discount on purchases over $1000 dollars.
Many people, in writing tests to express this policy, would create a test
that says: "If the customer is preferred, and the purchase amount is $1000
dollars, offer the discount." This test is erroneous. How will we detect
that error if all we have is the tests?

So I don't think tests (necessarily) express the same idea as the
requirements.

Another example. I have in mind a specific mathematical function that I
could describe reasonably well with words. But instead of describing the
function, I'll offer an incomplete set tests (the number on the left is the
input, the number on the right is the correct output):

1 -> 0
2 -> 1
3 -> 7
4 -> 2
5 -> 5
6 -> 8
7 -> 16
8 -> 3
9 -> 19
10 -> 6
11 -> 14
12 -> 9
13 -> 9
14 -> 17
15 -> 17
16 -> 4
17 -> 12
18 -> 20
19 -> 20
20 -> 7
21 -> 7
22 -> 15
23 -> 15
24 -> 10
25 -> 23
26 -> 10
27 -> 111
28 -> 18

Now: What's the requirement? That is, what's the function being calculated?
What's the correct value for an input of 29? How many additional tests would
you need to feel confident that the system gave the correct result for
inputs up to 1,000,000?

Dale

--
Dale Emery
Consultant to software teams and leaders
Web: http://dhemery.com
Weblog: http://cwd.dhemery.com

Re: Re: Can we always pre-prepare automated regression tests?

2009-12-05T12:39:09Z

Hi Steven,

Ultimately, we have to choose between:

> 1. Ambiguous requirements that are open to interpretation and therefore
> impossible to satisfy with certainty. (Yeah, you can say when the customer
> finally blesses it, but if I had a dollar for every time a waterfall
> customer came back and said it did not meet a spec.)
> 2. Truly precise requirements - this leads to precision tantamount to
> programming, but without feedback because it does not produce executable
> code to test. It creates huge overhead and makes changes very expensive.
> 3. Have concrete examples stand in the for the requirements. When 2 + 2 =4
> and -1 + 1 = 0 work, but 3 + 5 = 8 does not, we add the "requirement" 3 + 5
> = 8. If this sequence of examples does not stabilize relatively quickly
> into the requirement we really wanted, we have the wrong developers.
>
> Am I missing some other feasible choices?
>

Yes: Reasonably precise statements of requirements /accompanied by/ examples
that serve as tests.

As numerous people have pointed out, if the requirements are separate from
the tests, there's little chance of keeping them consistent with each other.
But we could keep each requirement in the same file with the tests that
exemplify it.

I haven't yet seen any tools that can parse a set of test files, extract the
requirements statements from them, and compose them into a requirements
document (with or without tests) on demand. But certainly such a tool is
well within our ability these days. (Ah, well, a boy can dream...)

Dale

--
Dale Emery
Consultant to software teams and leaders
Web: http://dhemery.com
Weblog: http://cwd.dhemery.com

Re: Re: Can we always pre-prepare automated regression tests?

2009-12-05T12:29:05Z

Hi Melody,

My opinion is that requirement documents are necessary, especially for
> larger organizations. Remember these documents are not only shared with QA
> and Development but are also used by technical writers, business analysts,
> account managers, marketing, sales, and PMs (project, program, and product
> managers). They provide an easy to understand way to provide this
> information to all teams. Not all of these people will have time, skill, or
> capability to access and read automated tests.
>

Imagine this scenario: For some software system, we have written a
reasonably comprehensive set of automated tests. Each test file includes not
only the automated tests for a given feature, but also a clear, concise
statement of the requirement being tested by those tests. Further, we have a
tool that can read the test files and display them on demand on a web page.
The tool allows users to display the requirements, the tests, or both.
Anyone who needs access to the requirements can visit the web site and see
them.

In that scenario, would you still want to specify all of the requirements in
a single, separately maintained document?

Dale

--
Dale Emery
Consultant to software teams and leaders
Web: http://dhemery.com
Weblog: http://cwd.dhemery.com

Re: Re: Can we always pre-prepare automated regression tests?

2009-12-05T12:18:41Z

Hi Mark,

I've been saying for years that we have to stop calling them tests. They're
> not tests. They are requirements specification. If the system is in state A,
> and it receives an input B, it returns C, and leaves the system in state D.
> This is a requirement.
>

I don't agree that this is always true, or even often true. A requirement
might be a statement of business policy. E.g. Preferred customers receive a
10 percent discount on each purchase over $1000. Or a statement of the
system's responsibility for a business policy: The system shall apply a 10
percent discount to each purchase by a preferred customer totalling over
$1000.

No matter how many examples you automate as tests, those tests are not the
requirement. The requirement is that the system enact this business policy.

For the sake of argument, let's assume that a requirement could be
> documented in an unambiguous, complete form. Now, don't we still want to
> convert this requirement into a test, preferably an automated test? I'd say
> yes.
>

Unless you could find a way to execute it directly. In fact, I'd see that as
one test of ambiguity: Could I automate it as is without adding information
or detail?

> Now we have to translate the requirement into a test, which is not only
> additional work, but introduces the possibility of a translation error.
>

Translating from one form to another gives a great opportunity to test your
understanding of the requirement, and whether the requirement was really
unambiguous after all.

I use the business policy above as an example in my workshops. I ask people
to write tests for the requirement. At some point, we always get to this
test: A preferred customer makes a purchase of $1000 dollars. Do we offer
the discount? Inevitably, some participants say yes, and some say no.

When that happens, I note the differing understandings, and read the policy
again. If people still disagree, I read the statement one more time,
emphasizing the word /over/. At that point, the people who were said we
should offer the discount change their minds.

So was the requirement stated ambiguously? I dunno. I do know that people
often understand it differently, and that translating the requirement into
tests -- as a group -- gives us a chance to detect those differing
understandings.

It seems unlikely to me that the business folks who created the policy
created the policy in terms of a half dozen examples. More likely, they
wanted to offer incentives for people to purchase from them rather than from
competitors. So they created a "preferred customer" status, and offered a
discount. They further offered the incentive to make large purchases by
putting a limit on which purchases receive the discount. At some point, they
may indeed walk through a number of examples, but they /started/ with a
statement of business policy, designed to achieve some business purpose
(large purchases by repeat customers).

So even if the software team receives only the tests, /somebody/ started
with a policy. If the intent of the requirement is to implement a business
policy, is it waste to tell the team what the policy is?

And if the requirement started as a business policy, then somebody somewhere
had to translated it into tests. And as my workshop example shows, it's easy
to make an error in that translation. So I like to do the translation in a
group, and I like to involve testers in the process, because good testers
are experts at detecting and exploiting ambiguity.

In another message, I'll offer a few examples of tests from which it is
difficult to guess what policy they exemplify.

So, as Steven states, we should simply start writing human-readable,
> executable requirements in the form of "tests", and dispense with the
> lengthy Word documents.
>

Lengthy Word documents are only one alternative.

Another: Include a concise English statement of the requirement in the test
file along with the automated tests.

Another: Write the tests for each story on the back of the story card, or on
front, or on a green card stapled to the story card.

Dale

--
Dale Emery
Consultant to software teams and leaders
Web: http://dhemery.com
Weblog: http://cwd.dhemery.com

Re: Re: Can we always pre-prepare automated regression tests?

2009-12-05T10:46:36Z

On Sat, Dec 5, 2009 at 10:34 AM, Matthew <matt.heusser@...> wrote:

>
>
>
> --- In agile-testing@..., George Dinwiddie <lists@...> wrote:
> >
> > Steven,
> >
> > "The software does not always do what we want it to do and we have
> > identified the following places where it falls short of our expectations."
> >
> > I like that! In fact, I'd like to drop using the word "requirements."
> > I think it leads us astray.
> >
> > - George
> >
>
> Replacing the term "requirements" with something more flexible ("desirements"?) -- I'm all for that. Good show, Man. Jolly good show(*).
>

Thanks, but using "desirements" at the customer-level does not really
lead anywhere unless we turn collaboratively turn "desirements" into
concrete, testable success criteria for each iteration so that:
- the team has concrete criteria to develop to, and
- everybody has concrete criteria to measure progress with and to
plan/steer the project with,

I like to call these concrete criteria "requirements". When we
discover through customer feedback or exploratory testing that meeting
these concrete requirements still leaves the "desirements" unfulfilled
or incomplete, I like to say we have discovered missing
"requirements", as opposed to bugs or defects. (Bugs or defects are
when the software does not meet the agreed upon concrete criteria).
Then, instead of panicking about have not achieved the goals of the
iteration, scrambling to try to do so at the last minute and creating
all sorts of chaos, we just schedule those missing requirements for
future iterations (to be prioritized by the customer).

We certainly should also look back on how those "requirements" were
missed and what the impact really was. If having not missed those
additional concrete criteria would have somehow allowed us to get more
work done (and it frequently would not have), we should figure out how
to avoid making that mistake in the future.

Steven Gordon

> regards,
>
> Matthew Heusser
> Blog: http://stpcollaborative.com/matt
>
> (*) - I just watched "A Christmas Carol" with the family and I'm in a Dickenish mood, ok?
>
>

------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/agile-testing/

<*> Your email settings:
Individual Email | Traditional

<*> To change settings online go to:
http://groups.yahoo.com/group/agile-testing/join
(Yahoo! ID required)

<*> To change settings via email:
agile-testing-digest@...
agile-testing-fullfeatured@...

<*> To unsubscribe from this group, send an email to:
agile-testing-unsubscribe@...

<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/

Re: Can we always pre-prepare automated regression tests?

2009-12-05T09:34:55Z

--- In agile-testing@..., George Dinwiddie <lists@...> wrote:

>
> Steven,
>
> "The software does not always do what we want it to do and we have
> identified the following places where it falls short of our expectations."
>
> I like that! In fact, I'd like to drop using the word "requirements."
> I think it leads us astray.
>
> - George
>

Replacing the term "requirements" with something more flexible ("desirements"?) -- I'm all for that. Good show, Man. Jolly good show(*).

regards,

Matthew Heusser
Blog: http://stpcollaborative.com/matt

(*) - I just watched "A Christmas Carol" with the family and I'm in a Dickenish mood, ok?

Re: Can we always pre-prepare automated regression tests?

2009-12-05T09:04:59Z

--- In agile-testing@..., "woynam" <woyna@...> wrote:
>
>
> Doesn't that simply mean that you missed a requirement?
>
...
> I would say that numbers 3 - 5 are areas where
>exploratory testing is useful, since it's primarily
>uncovering things that were missed in the specifications.
>However, once these types of defects are uncovered,
>the specification should be updated.
>

So I see a direct contradiction between this position (we need a comprehensive spec), well, the Agile Manifesto.

My life experience has been that when we find a contradiction, we need to go back and re-examine our assumptions.

Earlier this year I wrote long essay that covers this issue(1 and 2); you can find it in chapter 16 of "Beautiful Testing" by O'Reilly.

You can download my chapter (free!) on-line; a creative commons derivative with attribution license applies:

http://www.xndev.com/downloads/Beautiful_Testing_ch16.pdf

regards,

Matthew Heusser
Blog: http://blogs.stpcollaborative.com/matt

(1) - Short answer - if, by exploratory testing, I find a defect in tab order, I am unlikely to create an automated test for that. This is not /comprehensive/, but it gets the software into the hands of our customers at a reasonable speed. This conforms to the agile-manifesto. Contradiction solves?

(2) - Longer Answer - the decision to automate a test is likely based on a combination of factors: How critical is the feature, how often do the customers use it, will it regress, is it possible that it regresses in a way the "test automation" is to slow to detect, how expensive is it to automate, is it possible that a change that does not introduce a defect is registered falsely as a defect, what will the maintenance cost be, and so on. Saying "automate everything" trivializes this decision, trivializes the role of testing, and, in many cases, fails to acknowledge that certain types of critical testing work /even exist/. I find it said often by people who haven't actually studied testing in much depth; so much so that I wrote an essay on the subject. O'Reilly put it in a book and gave the book the subtitle "Leading Professionals Reveal How They Improve Software."

Re: Re: Can we always pre-prepare automated regression tests?

2009-12-04T17:28:41Z

Steven,

"The software does not always do what we want it to do and we have
identified the following places where it falls short of our expectations."

I like that! In fact, I'd like to drop using the word "requirements."
I think it leads us astray.

- George

Steven Gordon wrote:

>
>
> Not Mark, but I indeed believe that there is an important role for both
> customer feedback and exploratory testing.
>
> However, I believe that feedback should not be expressed in the form of
> "you did not fulfil the requirements you said you fulfilled", but
> instead if the form of "the software does not always do what we want it
> to do and we have identified the following places where it falls short
> of our expectations".
>
> Then the customer(s), developers and testers work together to translate
> those dissatisfactions into "requirements", i.e., concrete examples that
> describe where the software does not do what it is required to do.
>
> Then we implement those new requirements, and repeat until there are no
> dissatisfactions.
>
> On Fri, Dec 4, 2009 at 3:00 PM, George Dinwiddie
> <lists@... <mailto:lists@...>> wrote:
>
>
>
> Mark,
>
> There will always be implicit requirements. There will usually be
> things that don't seem as right when experienced than they did when
> envisioned.
>
> The world is not so deterministic that everything can be explicitly
> specified. That said, if you don't want to do exploratory testing, it's
> your choice.
>
> - George
>
>
>
> woynam wrote:
> > Doesn't that simply mean that you missed a requirement? What did the
> > customer find that was unacceptable? Why was that undocumented?
> >
> > What the purpose of exploratory testing? Isn't it to uncover hidden
> > defects? If one is found, should one add a new requirement
> > specification to account for the behavior?
> >
> > If we look at the generally definition of a defect [Patton 2006]:
> >
> > 1. The software doesn't do something that the product specification
> > says it should do
> >
> > 2. The software does something that the product specification says it
> > shouldn't do
> >
> > 3. The software does something that the product specification doesn't
> > mention.
> >
> > 4. The software doesn't do something that the product specification
> > doesn't mention, but should
> >
> > 5. The software is difficult to understand, hard to use, slow, or in
> > the software tester's eyes, will be viewed by the end user as just
> > not right
> >
> > I would say that numbers 3 - 5 are areas where exploratory testing is
> > useful, since it's primarily uncovering things that were missed in
> > the specifications. However, once these types of defects are
> > uncovered, the specification should be updated.
> >
> > Mark
> >
> > --- In agile-testing@...
> <mailto:agile-testing%40yahoogroups.com>, George Dinwiddie
> <lists@...> wrote:
> >> Steven,
> >>
> >> I understand the points you make. But I also understand when people
> >> find that the software passes all the automated, but doesn't
> meet their
> >> expectations, and are disappointed.
> >>
> >> I think that well-considered automated regression tests can
> minimize
> >> those times, but there will always be a place for exploratory
> testing, also.
> >>
> >> - George
> >>
> >> Steven Gordon wrote:
> >>>
> >>> Unfortunately, even a requirement statement as simple as "I
> want to add
> >>> 2 numbers" is too ambiguous to tell whether or not it has been
> satisfied.
> >>>
> >>> Ultimately, we have to choose between:
> >>> 1. Ambiguous requirements that are open to interpretation and
> therefore
> >>> impossible to satisfy with certainty. (Yeah, you can say when the
> >>> customer finally blesses it, but if I had a dollar for every
> time a
> >>> waterfall customer came back and said it did not meet a spec.)
> >>> 2. Truly precise requirements - this leads to precision
> tantamount to
> >>> programming, but without feedback because it does not produce
> executable
> >>> code to test. It creates huge overhead and makes changes very
> expensive.
> >>> 3. Have concrete examples stand in the for the requirements.
> When 2 + 2
> >>> =4 and -1 + 1 = 0 work, but 3 + 5 = 8 does not, we add the
> "requirement"
> >>> 3 + 5 = 8. If this sequence of examples does not stabilize
> relatively
> >>> quickly into the requirement we really wanted, we have the
> wrong developers.
> >>>
> >>> Am I missing some other feasible choices?
> >>>
> >>> Of those choices, examples standing for requirements is by far
> the most
> >>> lean and agile approach.
> >>>
> >>> On Thu, Dec 3, 2009 at 2:47 PM, George Dinwiddie
> >>> <lists@... <mailto:lists@ <mailto:lists@>...>> wrote:
> >>>
> >>>
> >>>
> >>> Mark,
> >>>
> >>> An automated test isn't a requirement in the sense that it's what a
> >>> person who matters wants the system to do. At best, it's an
> example of
> >>> such a requirement.
> >>>
> >>> As an example, I may want my calculator to add any two numbers.
> I can
> >>> specify the tests that 2+2 = 4 and -1 + 1 = 0, but these don't
> fully
> >>> specify my requirement. It could still get other values wrong.
> And it
> >>> probably has a numeric maximum that fails my "any two numbers"
> >>> requirement, but which these tests don't verify.
> >>>
> >>> It's a fairly fine point, but I've seen hours of conversation
> rotate
> >>> around this particular circle.
> >>>
> >>> - George

--
----------------------------------------------------------------------
* George Dinwiddie * http://blog.gdinwiddie.com
Software Development http://www.idiacomputing.com
Consultant and Coach http://www.agilemaryland.org
----------------------------------------------------------------------

------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/agile-testing/

<*> Your email settings:
Individual Email | Traditional

<*> To change settings online go to:
http://groups.yahoo.com/group/agile-testing/join
(Yahoo! ID required)

<*> To change settings via email:
agile-testing-digest@...
agile-testing-fullfeatured@...

<*> To unsubscribe from this group, send an email to:
agile-testing-unsubscribe@...

<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/