"Why is QA Always the bottleneck?"

[Matthew-103]

---> Has been published by SearchSoftwareQuality:

http://searchsoftwarequality.techtarget.com/tip/0,289483,sid92_gci1367905_mem1,00.html?track=NL-516&ad=725158&asrc=EM_USC_9213491&uid=9125356

As always, I'm interested in your feedback.  If you'd like to contribute to
a future article, drop me a note.

regards,

--
Matthew Heusser,
Testing Blog: http://xndev.blogspot.com
SW Craft Blog: http://craftingsw.blogspot.com/
Twitter: http://www.twitter.com/mheusser

"It is not always advantageous to automate test cases ..."
-  The Selenium RC Documentation itself,
http://seleniumhq.org/docs/01_introducing_selenium.html

[sialbs]

Two things.
I'm working with an organization that has the same characteristics as the one you describe - it's called QA and really only does testing.  There are three levels of testing on the software done by developers before the QA folk touch it and there is a round of testing done entirely by the users after QA.  Despite their lofty title of QA, no one in the QA organization has any authority for policy or process over any of the other quality activities during software development.  However, they are held responsible for the quality of the result, and are labeled a bottleneck with whatever euphemism is popular, especially when deadlines and developer evaluation and pride are at stake.  Your point that QA processes go faster when the product has higher quality when it comes to QA is well taken and highlights a failure of development to take testing seriously.  In this case, as in many other organizations, developers know that there is a QA group who will test and who will get the blame when the product is not tested well (after all those guys are the Quality guys, not us) so there is no incentive for the developers to increase the quality of the product before dispatching it to QA.
This organization has no definition of exit criteria for any of the testing stages done by any group.  There also exists a complicated and time consuming process of defining whether defects are really defects or are enhancements, or didn't appear in the requirements, or should have been in the requirement because we developers really know what the customer wants, or whatever. This process is fraught with political overtones arising from who gets the blame for the defect.
We are in the process of establishing a wider range for QA - giving them the authority of a QA department rather than tester - and are starting with designating quality exit criteria or at least a guideline for designating project-specific exit criteria so that QA can indeed provide some assurance that quality exists at each stage.  Fortunately we are getting some upper management support from the business - development is resisting - because the organization has a real QA department for the rest of the production and operations.
The second point also revolves about the difference between QA and testers.  That discussion you mention that creates a collaboration between management and the QA people is an excellent approach and perhaps the best option.  One thing has to be added: numbers.  For a tester (the view of management of those who test where a QA function has not been elevated to its proper level) to suggest that releasing software early is risky may be a forgone conclusion to the tester, but the manager is going to view the statement in a certain context: this guy is a tester, of course he is going to say that. It's his job to test. He gets paid to test.
What management needs is some facts - that is, some dollars and sense justification.  "In three weeks of testing we have reduced the risk of releasing this product from 100% to 40%.  At 40% there is a likelihood that the product will fail for some reason once a day and be done an average of 20 minutes. Marketing estimates a loss of $X for every failure.  With another week of testing we can reduce the risk to 20% resulting in only $X risk.  What would you like to do?"  When the cost of failure is greater than the cost of reducing the risk of that failure, management will likely back off any bottleneck argument and gain more respect for QA because QA is now talking their terms.  However, when the cost of additional testing does not reduce the risk enough to offset that cost, then testing is indeed a bottleneck and the product has already passed the "good enough" stage and should be released.
Good article, Matt.



From: Matt Heusser
Sent: Monday, September 14, 2009 11:21 AM
To: agile-testing@... ; sw-improve
Subject: [agile-testing] "Why is QA Always the bottleneck?"


  ---> Has been published by SearchSoftwareQuality:

http://searchsoftwarequality.techtarget.com/tip/0,289483,sid92_gci1367905_mem1,00.html?track=NL-516&ad=725158&asrc=EM_USC_9213491&uid=9125356

As always, I'm interested in your feedback.  If you'd like to contribute to a future article, drop me a note.

regards,

--
Matthew Heusser,
Testing Blog: http://xndev.blogspot.com 
SW Craft Blog: http://craftingsw.blogspot.com/
Twitter: http://www.twitter.com/mheusser

"It is not always advantageous to automate test cases ..."
-  The Selenium RC Documentation itself, http://seleniumhq.org/docs/01_introducing_selenium.html





--------------------------------------------------------------------------------



No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 8.5.409 / Virus Database: 270.13.94/2367 - Release Date: 09/13/09 05:50:00

[George Dinwiddie]

Steve Blais wrote:
> What management needs is some facts - that is, some dollars and sense
> justification.  "In three weeks of testing we have reduced the risk of
> releasing this product from 100% to 40%.  At 40% there is a likelihood
> that the product will fail for some reason once a day and be done an
> average of 20 minutes. Marketing estimates a loss of $X for every
> failure.  With another week of testing we can reduce the risk to 20%
> resulting in only $X risk.  What would you like to do?"

How do you calculate these numbers?

  - George

--
  ----------------------------------------------------------------------
   * George Dinwiddie *                      http://blog.gdinwiddie.com
   Software Development                    http://www.idiacomputing.com
   Consultant and Coach                    http://www.agilemaryland.org
  ----------------------------------------------------------------------

[sialbs]

OK.

 

Assumption:  when management (or anyone) accuses the test team of being a bottleneck, what they are saying is "you are not providing a value for the time you are taking that justifies the time you are taking, therefore you are unjustifiably holding up progress - a bottleneck." When the test team is able to show that it is providing value then it proves that it is not a bottleneck.  Therefore.

 

Step 1:  determine the value of the product to the organization.

   Let's use a simple example of making modifications or enhancements to a web site.  Checking with sales administration we determine that the current website averages $240K in revenue a day. (actual figures are higher, but we'll round it to an easy number for the example).

 

Step 2: identify an understandable and quantifiable risk to the value.

   We'll use the obvious one: web site crashes.  We can check our availability calculations for this. MTBF does not really help, but MTTR gives us the information we need.  The MTTR is 42.28 so for the example we'll make it 30 minutes.  Therefore the quantifiable risk is $50K.

 

Step 3:  compute the cost of the testing

   Assume 3 testers on the team doing nothing but working on the System Under Test.  For argument's sake, their combined gross pay is 3000 (average gross pay of 52K) which means the fully loaded cost runs out about 6200/week.  

 

Now the challenge is to link the cost of the testing (disregarding the overhead costs of equipment, space, etc.) to the reduction of the risk.

 

A note about risk reduction.  Risk, by definition, is uncertainty. Testing is about removing the uncertainty associated with any software product. When the product is introduced to the testing group there is a 100% chance of defects in the product  or 100% uncertainty about the product's success. The job of testing is to remove that uncertainty.  When a test is executed (any form of test) it removes uncertainty whether the result is that no defect was found, the product works but is defective, or the product fails as a result of the test.  In any case, there is no further uncertainty and therefore no risk.  When we find a defect we no longer have a risk that the defect exists, we have a fact: it is there. Once a defect has been found, someone else - typically someone in management or authority - determines whether to fix it or not. Either way, the risk is eliminated.

 

Approach 1 - saturation

At any given point in time (the point at which the bottleneck accusation takes place for example) there is a finite set of requirements (use cases, user stories, items on the burn down list, etc.) which defines the product identified in Step 1 above.  Assuming change will take place, the discussion is about what we know now, and we cannot anticipate what change might take place. Given a finite set of product describers, an equally finite set of tests can be devised that will exercise the product at this point in time.

For the example, we'll make two assumptions, both of which are false in reality, but they help make the math easier:

  1.. each test of the product is equal in terms of satisfying product describers and uncertainty reduction, and
  2.. defects are equally distributed among all product describers
 

At the bottleneck discussion 50 % of the tests have been executed reducing the remaining risk to 50%.  The time to execute the remaining tests is independent of the time it took to execute the first 50%, but let's arbitrarily say that it will take another two weeks to complete the tests to remove all uncertainty in the product.  Assuming that in the first 50% of tests the testing team found 5 severe defects (web crashers), then mathematically at this point the organization faces a risk that there are another 5 remaining and the quantifiable risk of that is $250K. Since the cost to identify those risks (and ostensibly get them fixed) is $12,400, there is no bottleneck; it makes good business sense to continue to test.

 

Keep in mind that when the defects are fixed the uncertainty goes back to 100% until we prove through testing that the defects have been removed.  

 

The approach is simplified for example's sake, but the approach is still valid.

 

OR

Approach 2 : product risk assessment

 

Someone, typically the business analyst or quality assurance, should have performed a product risk analysis.  This is different than the project risk analysis that the project manager performs.  Based on the analysis we have a vulnerability value.  We also are able to prioritize our quality assurance activities based on the identified risks to the organization due to failure of the product and remove the risk (uncertainty) through testing the high risk items.  We also create a measuring stick for the bottleneck discussion.  Based on the assigned value of the risk items compared to the overall vulnerability we can easily determine the percentage of risk remaining. Again, we can project the number of tests or parts of the product that need to be exercised to eliminate the specified uncertainty and, based on percent complete, reasonably state that the specified risk has been reduced by that percentage.  There are mathematical algorithms that can assign a quantifiable dollar value to the impact value of the risk assessment so that you can use dollars to demonstrate the value of continued testing. In the aggregate, this approach is weak, but probably will hold up in bottleneck discussions with management, especially if you take the precision to three decimal points.  

 

OR

Approach 3: problem solving

 

At the beginning of the project or initiative or whatever, someone, typically the business analyst gets answers to the following:

Is this the problem we need to solve? (statement of problem)

When "yes", what do you see as the result of providing a solution? (Statement of vision)

What do you need to see to prove that we have solved your problem? (acceptance criteria)

 

Based on the answer to the third question, a set of tests can be created to provide the proof requested.  Assuming that we start with a 100% risk that the product will not meet these tests, we can compute a percentage based on the number of those tests executed, each reducing the uncertainty. As we test, the percentage of uncertainty or risk is reduced.

 

However, there is the other side of the coin.  Suppose that the three testers are in end game testing when the accusation is leveled at them.  They are in retesting of bug-fixes and performing more extensive tests of the software.  The statistics show that the defect discovery rate is now at four defects per day and they have not unearthed a high severity defect in several weeks. This means that the team's position that more testing is necessary, assuming the system has been exercised completely at least once, becomes less defensible.  Management's return for the $6200 for an extra week's testing does not include any decreased risk to the effectiveness of the product. In this case, testing might well be considered a bottleneck: the cost of testing is greater than the value that the testing returns to the organization.

 

Further complications arise when marketing, which as we know never lies, predicts that the organization stands to gain $50K worth of new revenue for every day that the web site is operational.  Now the quantifiable impact increases significantly, but so does the impact of testing one more day to find that Elusive Defect that Justifies Your Existence (EDJYE).  The bottleneck designation may well be foisted on the testers unless evidence or algorithms can be proffered to show risk that offsets that marketing-guaranteed $50K per day. Or we can make a good argument for a threat to intangible benefits such as reputation.

 

 

 



From: George Dinwiddie
Sent: Thursday, September 17, 2009 9:29 AM
To: agile-testing@...
Subject: Re: QRe: [agile-testing] "Why is QA Always the bottleneck?"


  Steve Blais wrote:
> What management needs is some facts - that is, some dollars and sense
> justification. "In three weeks of testing we have reduced the risk of
> releasing this product from 100% to 40%. At 40% there is a likelihood
> that the product will fail for some reason once a day and be done an
> average of 20 minutes. Marketing estimates a loss of $X for every
> failure. With another week of testing we can reduce the risk to 20%
> resulting in only $X risk. What would you like to do?"

How do you calculate these numbers?

- George

--
----------------------------------------------------------
* George Dinwiddie * http://blog.gdinwiddie.com
Software Development http://www.idiacomputing.com
Consultant and Coach http://www.agilemaryland.org
----------------------------------------------------------






--------------------------------------------------------------------------------



No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 8.5.409 / Virus Database: 270.13.102/2377 - Release Date: 09/16/09 17:49:00

[Lisa Crispin-2]

I'm curious, since you have posted this on the agile-testing list, in what
sense is your organization agile? Is there a reason to not integrate the
testers in the "QA" department into the development teams, where they could
add value on the front end, instead of treating testing as a phase?

It's hard in the agile world to get statistics for the effects of "QA",
since it many aspects of agile that improve quality, and allow frequent
releases of valuable functionality. You can't really separate out testing
activities, they are too tightly integrated into development.

I know of agile teams that have a separate testing department and
post-development testing for a variety of reasons (embedded software,
needing to test with a vast number of other systems), but they still had
testers working together with the dev team during the development process.
-- Lisa

On Thu, Sep 17, 2009 at 7:27 AM, Steve Blais <sialbs@...> wrote:

>
>
> Two things.
> I'm working with an organization that has the same characteristics as the
> one you describe - it's called QA and really only does testing.  There are
> three levels of testing on the software done by developers before the QA
> folk touch it and there is a round of testing done entirely by the users
> after QA.  Despite their lofty title of QA, no one in the QA organization
> has any authority for policy or process over any of the other quality
> activities during software development.  However, they are held responsible
> for the quality of the result, and are labeled a bottleneck with whatever
> euphemism is popular, especially when deadlines and developer evaluation and
> pride are at stake.  Your point that QA processes go faster when the product
> has higher quality when it comes to QA is well taken and highlights a
> failure of development to take testing seriously.  In this case, as in many
> other organizations, developers know that there is a QA group who will test
> and who will get the blame when the product is not tested well (after all
> those guys are the Quality guys, not us) so there is no incentive for the
> developers to increase the quality of the product before dispatching it to
> QA.
> This organization has no definition of exit criteria for any of the testing
> stages done by any group.  There also exists a complicated and time
> consuming process of defining whether defects are really defects or are
> enhancements, or didn't appear in the requirements, or should have been in
> the requirement because we developers really know what the customer wants,
> or whatever. This process is fraught with political overtones arising from
> who gets the blame for the defect.
> We are in the process of establishing a wider range for QA - giving them
> the authority of a QA department rather than tester - and are starting with
> designating quality exit criteria or at least a guideline for designating
> project-specific exit criteria so that QA can indeed provide some assurance
> that quality exists at each stage.  Fortunately we are getting some upper
> management support from the business - development is resisting - because
> the organization has a real QA department for the rest of the production and
> operations.
> The second point also revolves about the difference between QA and
> testers.  That discussion you mention that creates a collaboration between
> management and the QA people is an excellent approach and perhaps the best
> option.  One thing has to be added: numbers.  For a tester (the view of
> management of those who test where a QA function has not been elevated to
> its proper level) to suggest that releasing software early is risky may be a
> forgone conclusion to the tester, but the manager is going to view the
> statement in a certain context: this guy is a tester, of course he is going
> to say that. It's his job to test. He gets paid to test.
> What management needs is some facts - that is, some dollars and sense
> justification.  "In three weeks of testing we have reduced the risk of
> releasing this product from 100% to 40%.  At 40% there is a likelihood that
> the product will fail for some reason once a day and be done an average of
> 20 minutes. Marketing estimates a loss of $X for every failure.  With
> another week of testing we can reduce the risk to 20% resulting in only $X
> risk.  What would you like to do?"  When the cost of failure is greater than
> the cost of reducing the risk of that failure, management will likely back
> off any bottleneck argument and gain more respect for QA because QA is now
> talking their terms.  However, when the cost of additional testing does not
> reduce the risk enough to offset that cost, then testing is indeed a
> bottleneck and the product has already passed the "good enough" stage and
> should be released.
> Good article, Matt.
>
>
>  *From:* Matt Heusser <matt.heusser@...>
> *Sent:* Monday, September 14, 2009 11:21 AM
> *To:* agile-testing@... ; sw-improve<SW-Improve@...>
> *Subject:* [agile-testing] "Why is QA Always the bottleneck?"
>
>
>
> ---> Has been published by SearchSoftwareQuality:
>
>
> http://searchsoftwarequality.techtarget.com/tip/0,289483,sid92_gci1367905_mem1,00.html?track=NL-516&ad=725158&asrc=EM_USC_9213491&uid=9125356
>
> As always, I'm interested in your feedback.  If you'd like to contribute to
> a future article, drop me a note.
>
> regards,
>
> --
> Matthew Heusser,
> Testing Blog: http://xndev.blogspot.com
> SW Craft Blog: http://craftingsw.blogspot.com/
> Twitter: http://www.twitter.com/mheusser
>
> "It is not always advantageous to automate test cases ..."
> -  The Selenium RC Documentation itself,
> http://seleniumhq.org/docs/01_introducing_selenium.html
>
> ------------------------------
>
>
> No virus found in this incoming message.
> Checked by AVG - www.avg.com
> Version: 8.5.409 / Virus Database: 270.13.94/2367 - Release Date: 09/13/09
> 05:50:00
>  
>

--
Lisa Crispin
Co-author with Janet Gregory, _Agile Testing: A Practical Guide for Testers
and Agile Teams_ (Addison-Wesley 2009)
http://lisacrispin.com

[George Dinwiddie]

Hi, Steve,

Bear with me on this.  I really appreciate your extended response, and
I'm sure you're telling me how it's actually done in your organization.
  This sort of quantification has always baffled me, partly because I
think it makes some assumptions that I don't hold.  The questions I have
don't reflect on you, but on my understanding of the process you describe.

Steve Blais wrote:
> Step 1:  determine the value of the product to the organization.
>
>    Let’s use a simple example of making modifications or enhancements to
> a web site.  Checking with sales administration we determine that the
> current website averages $240K in revenue a day. (actual figures are
> higher, but we’ll round it to an easy number for the example).

I think there's more value than just sales, but I'll accept that as a
first-order approximation of the value.

> Step 2: identify an understandable and quantifiable risk to the value.
>
>    We’ll use the obvious one: web site crashes.  We can check our
> availability calculations for this. MTBF does not really help, but MTTR
> gives us the information we need.  The MTTR is 42.28 so for the example
> we’ll make it 30 minutes.  Therefore the quantifiable risk is $50K.

So, you're saying that if the site crashes, it takes 42.28 minutes to
bring it back up (on average)?

Using the rounded figure of 30 minutes, that's 1/48th of a day.  That
seems to be $5K/crash, to me.  Is it going to crash 10 times?

But, it seems to me that I (as the hypothetical person making decisions)
don't want to know the cost per crash, per se.  If I release quarterly,
then for this release I want to know the cost per quarter.  Therefore,
MTBF /is/ important to me.  If it crashes once a day, that's a lot worse
than if it crashes once a quarter.  Or if the MTBF is 365 days, then
likely it won't crash at all before the next release.

Another problem for me is that I don't know how to predict MTBF and MTTR
for software systems.  I've seen MTBF formulas for hardware systems, but
they involved quite a bit of handwaving, and they only considered the
components of the system, now how well it was designed.  I know from
experience that a single poor design decision can result in heat that
results in a much lower MTBF.  In one case it was a rectifier that
couldn't actually handle the rated load that the datasheet said.  It
would heat up and unsolder itself from the board.  The MTBF of the unit
was in hours rather than the years one would expect.

Yet another problem for me is that crashing is not the only, or even
perhaps the biggest problem.  I worry about other things such as
computing the wrong answer, being confusing and driving away customers,
one part of the site just not working correctly in certain
circumstances, and things like that.  I know even less how to quantify
these things.

> Step 3:  compute the cost of the testing
>
>    Assume 3 testers on the team doing nothing but working on the System
> Under Test.  For argument’s sake, their combined gross pay is 3000
> (average gross pay of 52K) which means the fully loaded cost runs out
> about 6200/week.

OK

> Now the challenge is to link the cost of the testing (disregarding the
> overhead costs of equipment, space, etc.) to the reduction of the risk.
>
> A note about risk reduction.  Risk, by definition, is uncertainty.
> Testing is about removing the uncertainty associated with any software
> product. When the product is introduced to the testing group there is a
> 100% chance of defects in the product  or 100% uncertainty about the
> product’s success.

This seems to be overstating the case.  It seems to suggest that no one,
other than the testing group, makes any determination of value.  The
100% chance of defects could be a tautology, as there will be cases
where two people disagree on the desired behavior.  Stating there's a
100% chance of the product crashing would suggest that /no one/ had ever
run it.  Even then you might get lucky.

> The job of testing is to remove that uncertainty.  
> When a test is executed (any form of test) it removes uncertainty
> whether the result is that no defect was found, the product works but is
> defective, or the product fails as a result of the test.  In any case,
> there is no further uncertainty and therefore no risk.  When we find a
> defect we no longer have a risk that the defect exists, we have a fact:
> it is there. Once a defect has been found, someone else – typically
> someone in management or authority – determines whether to fix it or
> not. Either way, the risk is eliminated.

The uncertainty and the risk are greatly reduced, granted.

> Approach 1 – saturation
>
> At any given point in time (the point at which the bottleneck accusation
> takes place for example) there is a finite set of requirements (use
> cases, user stories, items on the burn down list, etc.) which defines
> the product identified in Step 1 above.  Assuming change will take
> place, the discussion is about what we know now, and we cannot
> anticipate what change might take place. Given a finite set of product
> describers, an equally finite set of tests can be devised that will
> exercise the product at this point in time.

Ah, I have trouble with a the finite set of product describers.  There
is a finite set of /explicit/ product describers, but an infinite set of
/implicit/ ones.  And some of those describers, even the explicit ones,
may require an infinite set of tests to evaluate.  Give the describer,
"the product should not crash in response to any user input," then we're
in trouble.  There's an infinite number of things a user might do.

> For the example, we’ll make two assumptions, both of which are false in
> reality, but they help make the math easier:
>
>    1. each test of the product is equal in terms of satisfying product
>       describers and uncertainty reduction, and
>    2. defects are equally distributed among all product describers
>
> At the bottleneck discussion 50 % of the tests have been executed
> reducing the remaining risk to 50%.  The time to execute the remaining
> tests is independent of the time it took to execute the first 50%, but
> let’s arbitrarily say that it will take another two weeks to complete
> the tests to remove all uncertainty in the product.  Assuming that in
> the first 50% of tests the testing team found 5 severe defects (web
> crashers), then mathematically at this point the organization faces a
> risk that there are another 5 remaining and the quantifiable risk of
> that is $250K. Since the cost to identify those risks (and ostensibly
> get them fixed) is $12,400, there is no bottleneck; it makes good
> business sense to continue to test.

Yes, it makes the math easier.  It also makes me uncomfortable.  It
leans heavily on the finite number of tests.

For example, if my testers had planned twice the number of tests, then
we're only 25% through them and now there are suddenly 15 more severe
defects.  Or if we discard a number of tests, then suddenly we've got
fewer defects remaining.  This leads to the unfortunate conclusion that
deciding to test more increases the number of defects in the product,
and testing less decreases the number of defects.  Assuming the tests
are good ones and not redundant, it is clearly false that less testing
results in less remaining risk.

And even if every planned test is executed and passes, the risks are not
reduced to zero.

> Keep in mind that when the defects are fixed the uncertainty goes back
> to 100% until we prove through testing that the defects have been removed.
>
> The approach is simplified for example’s sake, but the approach is still
> valid.

I must say that it doesn't seem valid to me.  Instead, it leaves me in a
VERY uncomfortable position.  While I agree with the sentiment that
testing resolves uncertainty, and that uncertainty hides risks, and that
risks have financial costs related to likelihood and consequences, and
that more testing, while costing both time and dollars, can
statistically reduce costs due to realized risks--the math has so many
bogus assumptions to make the calculation tractable that the results are
no longer meaningful.  They're also relatively easily manipulated in
many ways, just by changing a few assumptions.

> OR
>
> Approach 2 : product risk assessment
>
> Someone, typically the business analyst or quality assurance, should
> have performed a product risk analysis.  This is different than the
> project risk analysis that the project manager performs.  Based on the
> analysis we have a vulnerability value.  We also are able to prioritize
> our quality assurance activities based on the identified risks to the
> organization due to failure of the product and remove the risk
> (uncertainty) through testing the high risk items.  We also create a
> measuring stick for the bottleneck discussion.  Based on the assigned
> value of the risk items compared to the overall vulnerability we can
> easily determine the percentage of risk remaining. Again, we can project
> the number of tests or parts of the product that need to be exercised to
> eliminate the specified uncertainty and, based on percent complete,
> reasonably state that the specified risk has been reduced by that
> percentage.  There are mathematical algorithms that can assign a
> quantifiable dollar value to the impact value of the risk assessment so
> that you can use dollars to demonstrate the value of continued testing.
> In the aggregate, this approach is weak, but probably will hold up in
> bottleneck discussions with management, especially if you take the
> precision to three decimal points.

I can't evaluate this, as the math is not made explicit here.  I can,
however, understand the argument that high precision results from
complicated calculations can be very persuasive when people don't
realize that there is no discernible accuracy.

> OR
>
> Approach 3: problem solving
>
> At the beginning of the project or initiative or whatever, someone,
> typically the business analyst gets answers to the following:
>
> Is this the problem we need to solve? (statement of problem)
>
> When “yes”, what do you see as the result of providing a solution?
> (Statement of vision)
>
> What do you need to see to prove that we have solved your problem?
> (acceptance criteria)
>
> Based on the answer to the third question, a set of tests can be created
> to provide the proof requested.  Assuming that we start with a 100% risk
> that the product will not meet these tests, we can compute a percentage
> based on the number of those tests executed, each reducing the
> uncertainty. As we test, the percentage of uncertainty or risk is reduced.

This is the basic premise that scripted (whether manual or automated)
acceptance tests are sufficient to remove all uncertainty or risk.  I
suppose many people believe that, but I do not.  I think that there are
both questions that we can predetermine, and questions that will arise
later.

> However, there is the other side of the coin.  Suppose that the three
> testers are in end game testing when the accusation is leveled at them.  
> They are in retesting of bug-fixes and performing more extensive tests
> of the software.  The statistics show that the defect discovery rate is
> now at four defects per day and they have not unearthed a high severity
> defect in several weeks. This means that the team’s position that more
> testing is necessary, assuming the system has been exercised completely
> at least once, becomes less defensible.  Management’s return for the
> $6200 for an extra week’s testing does not include any decreased risk to
> the effectiveness of the product. In this case, testing might well be
> considered a bottleneck: the cost of testing is greater than the value
> that the testing returns to the organization.

This makes sense.

> Further complications arise when marketing, which as we know never lies,
> predicts that the organization stands to gain $50K worth of new revenue
> for every day that the web site is operational.  Now the quantifiable
> impact increases significantly, but so does the impact of testing one
> more day to find that Elusive Defect that Justifies Your Existence
> (EDJYE).  The bottleneck designation may well be foisted on the testers
> unless evidence or algorithms can be proffered to show risk that offsets
> that marketing-guaranteed $50K per day. Or we can make a good argument
> for a threat to intangible benefits such as reputation.

This makes it sound like a situation of competing interests.  And, it
doesn't sound like an Agile shop at all (even if they call themselves
that).  One of the important goals of Agile development is to get
everyone aligned for the organizations good, not to defend "my department."

I suspect that a simple graphing of defects uncovered (by severity of
consequences) over time will show a point of diminishing returns for
more testing.  While lacking 3 decimal points, this data is just as
accurate and likely easier to interpret.

I also know that the finite tests that you can predict are, for the most
part, amenable to automation.  Running this full set of tests should not
take a long time, so re-running after bug fixes is no longer a
bottleneck of significant proportions.  The bulk of remaining testing
can be of the infinite, exploratory variety that asks new questions
about the product.  If the finite set of predicted tests is /not/
automated, then testing /will/ be a bottleneck, whether justifiable or not.

The gains of automation are offset somewhat by the effort to automate
the tests.  One advantage of the automation is that it can be started
earlier, before the product is completed, and therefore doesn't usually
become a bottleneck because it's not serialized with the product
development time.  Another advantage is that they provide feedback to
development sooner, resulting in less effort spent writing code that
doesn't meet the explicit requirements.  Of course, the effort to
automate varies widely.  There are some great advances in tools to
reduce this effort.  There are also organizations that persist in making
things more difficult to automate (and, sometimes, to test manually)
than is necessary.

Thanks, again, Steve, for your description of how you quantify these
risks.  I don't believe the numbers, but I'm very glad to know how you
approach it.

  - George

--
  ----------------------------------------------------------------------
   * George Dinwiddie *                      http://blog.gdinwiddie.com
   Software Development                    http://www.idiacomputing.com
   Consultant and Coach                    http://www.agilemaryland.org
  ----------------------------------------------------------------------



------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/agile-testing/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/agile-testing/join
    (Yahoo! ID required)

<*> To change settings via email:
    mailto:agile-testing-digest@...
    mailto:agile-testing-fullfeatured@...

<*> To unsubscribe from this group, send an email to:
    agile-testing-unsubscribe@...

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

[sialbs]

Hi Lisa
I was actually responding wistfully to a good article by Matt who I place in the pantheon of Agile Practitioners and Pundits along with you and Ron and select others thus making the subject of "bottleneck" an agile consideration. Although I have to say that the concept of 'bottleneck' and especially of testers being a 'bottleneck' seems to be a non-agile consideration.  The longer response is to George and I thought it might be interesting to open a discussion on the overall value of testing since that is always a consideration - it is in the company I'm working with now - and how agile testing might increase that value.  
As for the company, it is a Fortune 100 company with over 600K employees and a large IT department concomitant with that size. The vast majority of its systems are on mainframes written in 3GLs (some 70 systems and 40 million lines of code) and supporting perhaps the largest transactional databases in the world They do have web-based systems on the outside.
I am trying to introduce some agility into the QA organization and many of the QA analysts want a more agile approach.  Currently all unit level, integration, and systems testing are done by developers.  QA performs a round of testing between systems testing and acceptance testing (done by users). Since the developers are doing developing and testing, perhaps that is agile by your definition?
They have found that entrusting both sides - development and testing - to the same groups results in defects that slip through. As a result the concept of "a fresh set of eyes" brings about a separated QA team (fairly large - four locations and some off-shore testing) to provide different perspectives and different techniques to uncover different defects.  The developers use Junit in some of their testing. I haven't worked my way to the developers groups as yet, so I don't have details on their testing methods; that's a project for next year.  I've introduced FIT and Fitnesse to the QA group.   I have also been pushing more collaborative efforts, but the developers and development organization appears to be pushing back.  Because of past history and culture, combining the development group and the QA group is not a consideration. Under these circumstances, do you have any suggestions as to how to increase the agility of the QA team operating as a separate unit?  I think I have succeeded in getting testers involved in the design review process as a first step.
The overall intent of my late night mathematical treatise in response to George was to make the point that Ron has made over and over in his exhortations about the value of XP.  When the XP iteration is over and the customer determines that the value to be received from the next iteration does not exceed the cost of that iteration, the project stops.  The same holds true of Scrum and any iterative method which has any form of serious review at the end of each iteration to determine the work for the next iteration.  I'm applying that concept to testing (or any part of software development) as a way of measuring the value of testing. I suggested risk reduction (rather than raw defect count) as a way of determining the value.  I'm also suggesting in a larger sense (for discussion only) that when the discussion about whether to fix a defect or even whether a found anomaly is really a defect takes longer than the time to find it then perhaps the product is good enough and the value of further testing is questionable.  
For the most part this is probably an academic exercise since testing usually ends with the time clock.
Can you suggest a way of proving to Upper Management the value of testing other than with risk reduction?  This would help us convince the PTB (powers that be) that it would be cost-effective to move QA back in the SDLC and engage more with the developers (developers attitudes aside).



From: Lisa Crispin
Sent: Friday, September 18, 2009 6:50 PM
To: agile-testing@...
Subject: Re: QRe: [agile-testing] "Why is QA Always the bottleneck?"


  I'm curious, since you have posted this on the agile-testing list, in what sense is your organization agile? Is there a reason to not integrate the testers in the "QA" department into the development teams, where they could add value on the front end, instead of treating testing as a phase?

It's hard in the agile world to get statistics for the effects of "QA", since it many aspects of agile that improve quality, and allow frequent releases of valuable functionality. You can't really separate out testing activities, they are too tightly integrated into development.

I know of agile teams that have a separate testing department and post-development testing for a variety of reasons (embedded software, needing to test with a vast number of other systems), but they still had testers working together with the dev team during the development process.
-- Lisa



On Thu, Sep 17, 2009 at 7:27 AM, Steve Blais <sialbs@...> wrote:

   

  Two things.
  I'm working with an organization that has the same characteristics as the one you describe - it's called QA and really only does testing.  There are three levels of testing on the software done by developers before the QA folk touch it and there is a round of testing done entirely by the users after QA.  Despite their lofty title of QA, no one in the QA organization has any authority for policy or process over any of the other quality activities during software development.  However, they are held responsible for the quality of the result, and are labeled a bottleneck with whatever euphemism is popular, especially when deadlines and developer evaluation and pride are at stake.  Your point that QA processes go faster when the product has higher quality when it comes to QA is well taken and highlights a failure of development to take testing seriously.  In this case, as in many other organizations, developers know that there is a QA group who will test and who will get the blame when the product is not tested well (after all those guys are the Quality guys, not us) so there is no incentive for the developers to increase the quality of the product before dispatching it to QA.
  This organization has no definition of exit criteria for any of the testing stages done by any group.  There also exists a complicated and time consuming process of defining whether defects are really defects or are enhancements, or didn't appear in the requirements, or should have been in the requirement because we developers really know what the customer wants, or whatever. This process is fraught with political overtones arising from who gets the blame for the defect.
  We are in the process of establishing a wider range for QA - giving them the authority of a QA department rather than tester - and are starting with designating quality exit criteria or at least a guideline for designating project-specific exit criteria so that QA can indeed provide some assurance that quality exists at each stage.  Fortunately we are getting some upper management support from the business - development is resisting - because the organization has a real QA department for the rest of the production and operations.
  The second point also revolves about the difference between QA and testers.  That discussion you mention that creates a collaboration between management and the QA people is an excellent approach and perhaps the best option.  One thing has to be added: numbers.  For a tester (the view of management of those who test where a QA function has not been elevated to its proper level) to suggest that releasing software early is risky may be a forgone conclusion to the tester, but the manager is going to view the statement in a certain context: this guy is a tester, of course he is going to say that. It's his job to test. He gets paid to test.
  What management needs is some facts - that is, some dollars and sense justification.  "In three weeks of testing we have reduced the risk of releasing this product from 100% to 40%.  At 40% there is a likelihood that the product will fail for some reason once a day and be done an average of 20 minutes. Marketing estimates a loss of $X for every failure.  With another week of testing we can reduce the risk to 20% resulting in only $X risk.  What would you like to do?"  When the cost of failure is greater than the cost of reducing the risk of that failure, management will likely back off any bottleneck argument and gain more respect for QA because QA is now talking their terms.  However, when the cost of additional testing does not reduce the risk enough to offset that cost, then testing is indeed a bottleneck and the product has already passed the "good enough" stage and should be released.
  Good article, Matt.



  From: Matt Heusser
  Sent: Monday, September 14, 2009 11:21 AM
  To: agile-testing@... ; sw-improve
  Subject: [agile-testing] "Why is QA Always the bottleneck?"


   
  ---> Has been published by SearchSoftwareQuality:

  http://searchsoftwarequality.techtarget.com/tip/0,289483,sid92_gci1367905_mem1,00.html?track=NL-516&ad=725158&asrc=EM_USC_9213491&uid=9125356

  As always, I'm interested in your feedback.  If you'd like to contribute to a future article, drop me a note.

  regards,

  --
  Matthew Heusser,
  Testing Blog: http://xndev.blogspot.com 
  SW Craft Blog: http://craftingsw.blogspot.com/
  Twitter: http://www.twitter.com/mheusser

  "It is not always advantageous to automate test cases ..."
  -  The Selenium RC Documentation itself, http://seleniumhq.org/docs/01_introducing_selenium.html




------------------------------------------------------------------------------



  No virus found in this incoming message.
  Checked by AVG - www.avg.com
  Version: 8.5.409 / Virus Database: 270.13.94/2367 - Release Date: 09/13/09 05:50:00




--
Lisa Crispin
Co-author with Janet Gregory, _Agile Testing: A Practical Guide for Testers and Agile Teams_ (Addison-Wesley 2009)
http://lisacrispin.com






--------------------------------------------------------------------------------



No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 8.5.409 / Virus Database: 270.13.105/2380 - Release Date: 09/18/09 07:49:00

[Steven Gordon-2]

On Sat, Sep 19, 2009 at 2:07 AM, Steve Blais <sialbs@...> wrote:
>
>
>
> Can you suggest a way of proving to Upper Management the value of testing other than with risk reduction?  This would help us convince the PTB (powers that be) that it would be cost-effective to move QA back in the SDLC and engage more with the developers (developers attitudes aside).
>

Just integrate QA (which is more than just testing), as well as every
other software development specialization, into software development
so seamlessly that upper management just sees a professional software
development department, not the individual pieces to be micromanaged.

If upper management does not allow this, then you have to find a way
to tell the following to upper management (in more polite terms):

We are the professional software developers.  We take responsibility
to develop software in a professional manner.  Do you ask all the
other departments to justify their professional practices for
accomplishing their goals and responsibilities?

I, like George, would feel unprofessional cooking the numbers to
justify what you know is the professional way to develop software.

[Matthew-103]

--- In agile-testing@..., Steven Gordon <sgordonphd@...> wrote:
 
> Just integrate QA (which is more than just testing), as well
> as every other software development specialization, into
> software development so seamlessly that upper management
> just sees a professional software development department

I agree with your sentiment, Steve, and I have to say "just" is a bit of a tough way to start out the paragraph.  While I don't want to justify a victim mentality, I have to empathize with big, slow, old, corporations that have institutional inertia and move at a snail's pace.  

I did a blog post awhile ago in this area that a few people might find helpful, called "How to be a first-class citizen as a tester"

http://xndev.blogspot.com/2009/03/how-to-be-first-class-citizen-as-tester.html

regards,

--heusser

[Steven Gordon-2]

I agree this is difficult in many corporate cultures to do this.

However, when a company chooses to adopt agile, upper management must know
that this means it is agreeing to determining the what, but delegating the
how.  If they cannot agree to that, then the company is not really choosing
to adopt an agile approach.  If this is not made abundantly clear to upper
management, then whoever is leading the agile adoption effort is shirking
their responsibility.

If this has not been established as a precondition, then attempting to
achieve this understanding is still the more useful goal than attempting to
prove the unprovable.  The latter approach just turns into a huge
distraction that is unwinnable in the long run.

On Sat, Sep 19, 2009 at 5:39 PM, Matthew <matt.heusser@...> wrote:

>
>
> --- In agile-testing@... <agile-testing%40yahoogroups.com>,
> Steven Gordon <sgordonphd@...> wrote:
>
> > Just integrate QA (which is more than just testing), as well
> > as every other software development specialization, into
> > software development so seamlessly that upper management
> > just sees a professional software development department
>
> I agree with your sentiment, Steve, and I have to say "just" is a bit of a
> tough way to start out the paragraph. While I don't want to justify a victim
> mentality, I have to empathize with big, slow, old, corporations that have
> institutional inertia and move at a snail's pace.
>
> I did a blog post awhile ago in this area that a few people might find
> helpful, called "How to be a first-class citizen as a tester"
>
>
> http://xndev.blogspot.com/2009/03/how-to-be-first-class-citizen-as-tester.html
>
> regards,
>
> --heusser
>
>  
>

[sialbs]

Thanks for the considered and most likely time consuming response.  It's
spot on. My initial response was to Matt's suggestion of testers being
considered a bottleneck.  While the term "bottleneck" per se hasn't come up,
we have heard many variations of the accusation.  It boils down to "the
testers are holding up production looking for bugs that either don't exist
or are not important".
From this perspective I was postulating that providing the decision makers
with some proof or measurement that the system under test needs the
additional testing removes the imprimatur of "bottleneck".  Granted, it
would be nice if everyone got along cooperatively and that clearly
represents the ideal.  However, as long as different parts of the
organization are incentivized in ways that tend to be contradictory - e.g.
marketers need to get the changes out to sell more product with the new
release, especially since most likely they have promised customers the
product is coming, and testers and developers need to see that the product
meets an acceptable level of confidence or quality (not necessarily the same
thing) before shipping regardless of the time it takes. This is the same as
the old marketing - manufacturing divide we've had for decades.
I'll only respond to some of your comments.  The rest I am acknowledging as
correct and I agree to them or at a too detailed level to spend more time
discussing.

>
> For example, if my testers had planned twice the number of tests, then
> we're only 25% through them and now there are suddenly 15 more severe
> defects.  Or if we discard a number of tests, then suddenly we've got
> fewer defects remaining.  This leads to the unfortunate conclusion that
> deciding to test more increases the number of defects in the product,
> and testing less decreases the number of defects.  Assuming the tests
> are good ones and not redundant, it is clearly false that less testing
> results in less remaining risk.
>
> And even if every planned test is executed and passes, the risks are not
> reduced to zero.

Agreed.  We both agree that there can be an infinite number of tests
performed on any non-simplistic piece of software. (I don't know how one
would define "non-simplistic").  I'm using the term "finite" to refer to the
cap that is put on testing at some point by someone whether that cap be put
based on quality criteria or the time on the clock.  In other words, there
is always a finite amount of testing for any given piece of software.  The
challenge is to reduce the maximum amount of risk of failure with that
finite amount of testing.  The same holds true for risk level. Every test
executed reduces (or should reduce) the level of risk (or increase the level
of confidence) in a piece of software.

>
> I must say that it doesn't seem valid to me.  Instead, it leaves me in a
> VERY uncomfortable position.  While I agree with the sentiment that
> testing resolves uncertainty, and that uncertainty hides risks, and that
> risks have financial costs related to likelihood and consequences, and
> that more testing, while costing both time and dollars, can
> statistically reduce costs due to realized risks--the math has so many
> bogus assumptions to make the calculation tractable that the results are
> no longer meaningful.  They're also relatively easily manipulated in
> many ways, just by changing a few assumptions.

Yes.  True.  A first cut attempt as I made (very early in the AM) includes a
lot of assumptions, some of which can be eliminated with more information
and some can't.  The idea is to get testers or QA people to take the extra
time to quantify in some way the testing they are doing so that management
understands why additional time is being taken and thus remove the label of
"bottleneck".  I am dealing with testers this summer and fall at two
radically different organizations and yet there is a similarity in
complaints: management doesn't understand the value of testing and why we
need to spend the time.  Management thinks we are a necessary evil.
Management needs to be enlightened."  My response:  enlighten them. My
conversations with management do not expose a malicious bias against testers
but rather a total lack of understanding of what value testers bring to the
equation.  In these cases management's understanding of testing is created
by the development team management and that is not always unbiased to begin
with.  While marketing is showing management numbers to get a release,
testers are relying on either fear (if it's not tested it will break) or
altruism (clearly we want the highest quality product to go out the door).
I absolutely believe that collaborative efforts are the best way to overcome
this issue, as long as the collaborative efforts include the upper level
management who are making the final decisions.

> This is the basic premise that scripted (whether manual or automated)
> acceptance tests are sufficient to remove all uncertainty or risk.  I
> suppose many people believe that, but I do not.  I think that there are
> both questions that we can predetermine, and questions that will arise
> later.

Absolutely true.

> This makes it sound like a situation of competing interests.  And, it
> doesn't sound like an Agile shop at all (even if they call themselves
> that).  One of the important goals of Agile development is to get
> everyone aligned for the organizations good, not to defend "my
> department."
>
> I suspect that a simple graphing of defects uncovered (by severity of
> consequences) over time will show a point of diminishing returns for
> more testing.  While lacking 3 decimal points, this data is just as
> accurate and likely easier to interpret.

Exactly.  Of course the graphing approach or actually the approach of
determining quality or readiness for release based on a defect discovery
rate has met with derision in the ranks of testers ("just because the number
of defects found goes down doesn't mean there isn't a serious defect left."
"those numbers reflect badly on testers because the inference can be made
that the testers aren't working as hard." and so forth).  I think defect
discovery rate is an adequate measure of the improvement of quality in the
product over time.  Besides, management really likes graphs, especially
those that appear to point in the right direction. Even without three
decimal points.

>
> I also know that the finite tests that you can predict are, for the most
> part, amenable to automation.  Running this full set of tests should not
> take a long time, so re-running after bug fixes is no longer a
> bottleneck of significant proportions.  The bulk of remaining testing
> can be of the infinite, exploratory variety that asks new questions
> about the product.  If the finite set of predicted tests is /not/
> automated, then testing /will/ be a bottleneck, whether justifiable or
> not.
>
> The gains of automation are offset somewhat by the effort to automate
> the tests.  One advantage of the automation is that it can be started
> earlier, before the product is completed, and therefore doesn't usually
> become a bottleneck because it's not serialized with the product
> development time.  Another advantage is that they provide feedback to
> development sooner, resulting in less effort spent writing code that
> doesn't meet the explicit requirements.  Of course, the effort to
> automate varies widely.  There are some great advances in tools to
> reduce this effort.  There are also organizations that persist in making
> things more difficult to automate (and, sometimes, to test manually)
> than is necessary.
>

And I agree one hundred percent with your concept of automation.  I assumed
that Matt's original premise was that the testing group was being accused of
being a bottleneck despite having a large percentage of tests automated.
Automation is one of the things we are working on this summer at both
organizations.


> Thanks, again, Steve, for your description of how you quantify these
> risks.  I don't believe the numbers, but I'm very glad to know how you
> approach it.
>
>  - George

And thank you George for the wonderful points.  I learned much and that
makes the effort of posting all the more worth while.
=steve
 



------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/agile-testing/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/agile-testing/join
    (Yahoo! ID required)

<*> To change settings via email:
    mailto:agile-testing-digest@...
    mailto:agile-testing-fullfeatured@...

<*> To unsubscribe from this group, send an email to:
    agile-testing-unsubscribe@...

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

[sialbs]

Not cooking the numbers, Steve.  Simply taking the time to identify and record the numbers that are there inherently in the process rather than simply going on faith that management will see the light and realize the importance of testing.  Most likely management will come to that realization, but only after some major disaster occurs for which the testers will be blamed.
 The thing to keep in mind is that while we understand the professional approach to software development the VP of Marketing, who is the heir apparent to the CEO and has the Board's ear, does not. The VP Marketing is only concerned about the precipitous drop in sales that he perceives comes from slow delivery of new products from IT.



From: Steven Gordon
Sent: Saturday, September 19, 2009 12:47 PM
To: agile-testing@...
Subject: Re: QRe: [agile-testing] "Why is QA Always the bottleneck?"


  On Sat, Sep 19, 2009 at 2:07 AM, Steve Blais <sialbs@...> wrote:
>
>
>
> Can you suggest a way of proving to Upper Management the value of testing other than with risk reduction?  This would help us convince the PTB (powers that be) that it would be cost-effective to move QA back in the SDLC and engage more with the developers (developers attitudes aside).
>

Just integrate QA (which is more than just testing), as well as every
other software development specialization, into software development
so seamlessly that upper management just sees a professional software
development department, not the individual pieces to be micromanaged.

If upper management does not allow this, then you have to find a way
to tell the following to upper management (in more polite terms):

We are the professional software developers. We take responsibility
to develop software in a professional manner. Do you ask all the
other departments to justify their professional practices for
accomplishing their goals and responsibilities?

I, like George, would feel unprofessional cooking the numbers to
justify what you know is the professional way to develop software.





--------------------------------------------------------------------------------



No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 8.5.409 / Virus Database: 270.13.108/2383 - Release Date: 09/19/09 17:50:00

[Steven Gordon-2]

Does the financial department have to justify the need for accounting
standards and independent audits?  (Events over the last decade indicate
that that case may have been lost at many companies - it is a case that
should never even be entertained.)

The VP of Marketing can certainly be concerned and give feedback about
delivery times, but has just as much right to question the role of testing
as the developers do questioning the specifics of how Marketing decides what
features are needed, when they are needed and when to announce them (of
course, Marketing can be given feedback on what it does just not
micro-managed on their internal processes).

I am not questioning your cost model that tries to explain the relationship
between testing and risk, just stating that the actual numbers you feed into
that model are pure guesses and can easily be manipulated to get whatever
result you want.  If you present it as:
- if the numbers were A1, A2, ... An, the model would predict A,
- if the numbers were instead B1, B2, .. Bn, the model would instead predict
B,
- etc.
in order to depict the relationship between testing and risk, I would not
call it cooking.  But, if you provide just one set of inputs as if they are
known values, then you are cooking the numbers.

This case is between development and the VP of Development.  If the VP of
development buys it, he has to say to upper management only that his
department will take responsibility to deliver as quickly as they
responsibly can.  If the VP of development does not buy it, then you cannot
do agile software development or maybe even professionally responsible
software development.

On Sun, Sep 20, 2009 at 6:49 AM, Steve Blais <sialbs@...> wrote:

>
>
> Not cooking the numbers, Steve.  Simply taking the time to identify and
> record the numbers that are there inherently in the process rather than
> simply going on faith that management will see the light and realize the
> importance of testing.  Most likely management will come to that
> realization, but only after some major disaster occurs for which the testers
> will be blamed.
>  The thing to keep in mind is that while we understand the professional
> approach to software development the VP of Marketing, who is the heir
> apparent to the CEO and has the Board's ear, does not. The VP Marketing is
> only concerned about the precipitous drop in sales that he perceives comes
> from slow delivery of new products from IT.
>
>
>  *From:* Steven Gordon <sgordonphd@...>
> *Sent:* Saturday, September 19, 2009 12:47 PM
> *To:* agile-testing@...
> *Subject:* Re: QRe: [agile-testing] "Why is QA Always the bottleneck?"
>
>
>
> On Sat, Sep 19, 2009 at 2:07 AM, Steve Blais <sialbs@...> wrote:
>
> >
> >
> >
> > Can you suggest a way of proving to Upper Management the value of testing
> other than with risk reduction?  This would help us convince the PTB (powers
> that be) that it would be cost-effective to move QA back in the SDLC and
> engage more with the developers (developers attitudes aside).
> >
>
> Just integrate QA (which is more than just testing), as well as every
> other software development specialization, into software development
> so seamlessly that upper management just sees a professional software
> development department, not the individual pieces to be micromanaged.
>
> If upper management does not allow this, then you have to find a way
> to tell the following to upper management (in more polite terms):
>
> We are the professional software developers. We take responsibility
> to develop software in a professional manner. Do you ask all the
> other departments to justify their professional practices for
> accomplishing their goals and responsibilities?
>
> I, like George, would feel unprofessional cooking the numbers to
> justify what you know is the professional way to develop software.
>
> ------------------------------
>
>
> No virus found in this incoming message.
> Checked by AVG - www.avg.com
> Version: 8.5.409 / Virus Database: 270.13.108/2383 - Release Date: 09/19/09
> 17:50:00
>  
>

[George Dinwiddie]

Steve Blais wrote:
> I'll only respond to some of your comments.  The rest I am acknowledging as
> correct and I agree to them or at a too detailed level to spend more time
> discussing.

Likewise

> The idea is to get testers or QA people to take the extra
> time to quantify in some way the testing they are doing so that management
> understands why additional time is being taken and thus remove the label of
> "bottleneck".  I am dealing with testers this summer and fall at two
> radically different organizations and yet there is a similarity in
> complaints: management doesn't understand the value of testing and why we
> need to spend the time.  Management thinks we are a necessary evil.
> Management needs to be enlightened."  My response:  enlighten them. My
> conversations with management do not expose a malicious bias against testers
> but rather a total lack of understanding of what value testers bring to the
> equation.  In these cases management's understanding of testing is created
> by the development team management and that is not always unbiased to begin
> with.  While marketing is showing management numbers to get a release,
> testers are relying on either fear (if it's not tested it will break) or
> altruism (clearly we want the highest quality product to go out the door).
> I absolutely believe that collaborative efforts are the best way to overcome
> this issue, as long as the collaborative efforts include the upper level
> management who are making the final decisions.

Ah, yes.  It's been a few years since I've dealt with that sort of
environment.  The problems, and the arguments for more testing, seem to
derive in large part from having a testing phase at the end.  I've found
it more profitable to teach management that testing is not the caboose
of the train, but something that needs attention all the way through.
While change is slow, it seems to be an easier thing to sell than a
longer testing phase.

>> I suspect that a simple graphing of defects uncovered (by severity of
>> consequences) over time will show a point of diminishing returns for
>> more testing.  While lacking 3 decimal points, this data is just as
>> accurate and likely easier to interpret.
>
> Exactly.  Of course the graphing approach or actually the approach of
> determining quality or readiness for release based on a defect discovery
> rate has met with derision in the ranks of testers ("just because the number
> of defects found goes down doesn't mean there isn't a serious defect left."
> "those numbers reflect badly on testers because the inference can be made
> that the testers aren't working as hard." and so forth).  I think defect
> discovery rate is an adequate measure of the improvement of quality in the
> product over time.  Besides, management really likes graphs, especially
> those that appear to point in the right direction. Even without three
> decimal points.

Most managers don't understand three decimal points.  That's why they're
not engineers. ;-)

As for the testers, /nothing/ can assure that there isn't a serious
defect left.  Defect discovery rate /can/ be (but isn't always) an
adequate measure of improvement in quality.  It depends on the order in
which things are tested.  It behooves the testers, for the sake of their
own reputation, to order their testing such that it likely finds the
more serious problems first.  This not only helps the graph be more
useful, but helps protect their reputation from catastrophic bugs found
just after release.  Unfortunately, this requires some hard thinking.

Someday I'd like to experience Michael Bolton's presentation of James
Bach's Rapid Software Testing class.  I suspect that this class is
designed to teach some of this hard thinking in an experiential manner.
  Certainly some of the exercises I've done with Michael, from time to
time, have delved into this area.  It's great fun to open a bottle of
wine and sit down to one of Michael's testing challenges.

  - George

--
  ----------------------------------------------------------------------
   * George Dinwiddie *                      http://blog.gdinwiddie.com
   Software Development                    http://www.idiacomputing.com
   Consultant and Coach                    http://www.agilemaryland.org
  ----------------------------------------------------------------------



------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/agile-testing/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/agile-testing/join
    (Yahoo! ID required)

<*> To change settings via email:
    mailto:agile-testing-digest@...
    mailto:agile-testing-fullfeatured@...

<*> To unsubscribe from this group, send an email to:
    agile-testing-unsubscribe@...

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

[sialbs]

Hi Steve
I understand and am in agreement with the concept that everyone has the right to question everyone else in the organization.  In a company of over 400,000 employees spread around the world adherence to this concept might easily breed chaos.  While we might accept that a tester might have some good ideas that an MBA in marketing with 20 years experience might not have, it's hard to believe that someone in marketing has any idea of what testing is about much less provide any valuable feedback especially when the testing group and marketing group and psychologically as well as geographically disparate.

Thanks for the explanation and I now understand what you are saying about "cooking".  That was not my intent.  I am only trying to engender the same rigor in testers dealing with management that the testers apply to the systems they are testing.  

Your comments about how it should be does suggest a question though.  I'm meeting with some of the senior managers on the business side this week.  My approach has been to attempt to move control of testing to a more central body, namely the testers under QA.  As said before in my note to Lisa, the developers do the unit, integration and system testing and the users do the acceptance testing independently of QA.  QA does testing between systems and acceptance.  There are the usual fingers pointed as you can imagine and lines drawn in the sand and so forth.  
What I'm wondering is whether you might have a suggestion of how to present a pitch for more agile approaches considering that the bulk of their processes - in the tens of millions of transactions a day, operating 24x7 - are on backend mainframes with code written in 3GL and 4GL.  There are "pockets of agile" somewhere, but I haven't stumbled upon them as yet in the eight weeks I've been here.  Of course, I've only been dealing with the QA department so far.  I'd love to be able to have a stronger agile pitch to senior management, but am at a loss.  I'm focusing on repairing the communications breakdowns up and down the lines at this moment.  Note that in two of the locations I've been to so far there has been a similar refrain: the testers and developers should work it out and stop their complaining.  Underlying message:  there are plenty out there looking for work if you don't like it here.  
Any suggestions?
Thanks in advance.
=steve




From: Steven Gordon
Sent: Sunday, September 20, 2009 11:59 AM
To: agile-testing@...
Subject: Re: QRe: [agile-testing] "Why is QA Always the bottleneck?"


  Does the financial department have to justify the need for accounting standards and independent audits?  (Events over the last decade indicate that that case may have been lost at many companies - it is a case that should never even be entertained.)

The VP of Marketing can certainly be concerned and give feedback about delivery times, but has just as much right to question the role of testing as the developers do questioning the specifics of how Marketing decides what features are needed, when they are needed and when to announce them (of course, Marketing can be given feedback on what it does just not micro-managed on their internal processes).

I am not questioning your cost model that tries to explain the relationship between testing and risk, just stating that the actual numbers you feed into that model are pure guesses and can easily be manipulated to get whatever result you want.  If you present it as:
- if the numbers were A1, A2, ... An, the model would predict A,
- if the numbers were instead B1, B2, .. Bn, the model would instead predict B,
- etc.
in order to depict the relationship between testing and risk, I would not call it cooking.  But, if you provide just one set of inputs as if they are known values, then you are cooking the numbers.

This case is between development and the VP of Development.  If the VP of development buys it, he has to say to upper management only that his department will take responsibility to deliver as quickly as they responsibly can.  If the VP of development does not buy it, then you cannot do agile software development or maybe even professionally responsible software development.  



On Sun, Sep 20, 2009 at 6:49 AM, Steve Blais <sialbs@...> wrote:

   

  Not cooking the numbers, Steve.  Simply taking the time to identify and record the numbers that are there inherently in the process rather than simply going on faith that management will see the light and realize the importance of testing.  Most likely management will come to that realization, but only after some major disaster occurs for which the testers will be blamed.
   The thing to keep in mind is that while we understand the professional approach to software development the VP of Marketing, who is the heir apparent to the CEO and has the Board's ear, does not. The VP Marketing is only concerned about the precipitous drop in sales that he perceives comes from slow delivery of new products from IT.



  From: Steven Gordon
  Sent: Saturday, September 19, 2009 12:47 PM
  To: agile-testing@...
  Subject: Re: QRe: [agile-testing] "Why is QA Always the bottleneck?"


   
  On Sat, Sep 19, 2009 at 2:07 AM, Steve Blais <sialbs@...> wrote:


  >
  >
  >
  > Can you suggest a way of proving to Upper Management the value of testing other than with risk reduction?  This would help us convince the PTB (powers that be) that it would be cost-effective to move QA back in the SDLC and engage more with the developers (developers attitudes aside).
  >

  Just integrate QA (which is more than just testing), as well as every
  other software development specialization, into software development
  so seamlessly that upper management just sees a professional software
  development department, not the individual pieces to be micromanaged.

  If upper management does not allow this, then you have to find a way
  to tell the following to upper management (in more polite terms):

  We are the professional software developers. We take responsibility
  to develop software in a professional manner. Do you ask all the
  other departments to justify their professional practices for
  accomplishing their goals and responsibilities?

  I, like George, would feel unprofessional cooking the numbers to
  justify what you know is the professional way to develop software.



------------------------------------------------------------------------------



  No virus found in this incoming message.
  Checked by AVG - www.avg.com

  Version: 8.5.409 / Virus Database: 270.13.108/2383 - Release Date: 09/19/09 17:50:00







--------------------------------------------------------------------------------



No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 8.5.409 / Virus Database: 270.13.109/2384 - Release Date: 09/20/09 06:22:00

[Steven Gordon-2]

It seems like this organization has not adopted agile top-down, so my
previous comments are fully applicable to such an organization,

Top-down agile adoption cannot be successful in an organization that
believes it is doing just fine with its current approach.  The attitude that
there are lots of people waiting to fill the jobs of people who do not like
the way things are done here indicates that the organization as a whole is
far from ready to consider a different approach.

Given that, agile approaches can still work in isolated sub-organizations,
but the people who interface between an agile sub-organization and the rest
of the organization face the formidable challenge of living in both worlds.

Centralizing QA is a step away from agility. It may solve the most visible
problems today, but I believe it will always create worse problems in the
long run.  It adds extra communication overhead, increases the size of
feedback loops, and breaks down the responsibility for delivering
high-quality working software in a way such that nobody is really
responsible.

The cetnral argument at a non-agile organization for integrating QA into
each project team is that doing so now makes the team itself responsible for
deliveribng software.  This does not necessarily mean that the QA people on
the project team might not have also report to a centralized QA authority,
especially in an already matrixed organization, but the daily responsibility
is to collaborate with everybody else on the project team to deliver working
software early and often.  Again, this also includes other specializations
such as DBAs, BAs, documentors and trainers (although some may not have
enough work to be full-time members of the project team during the entire
life cycle).

Once the product owner(s) and development team agree to this, then the next
step is to move from waterfall-like approach to a leaner user-story-based
approach that allows delivery of working software every two weeks without
entire use cases being fully implemented in every delivery.  The deliveries
do not have to be deployable, but the rhythm of delivering software that
works and getting concrete feedback on that delivery convinces the business
people that the team is actually delivering, not twiddling their thumbs
creating designs on paper until just before the release date.  It also means
there will always be something to deliver on release date, even if a few
features have slipped, as opposed to the all-or-nothing delivery the company
may well be used to.

SteveG

On Mon, Sep 21, 2009 at 4:46 AM, Steve Blais <sialbs@...> wrote:

>
>
> Hi Steve
> I understand and am in agreement with the concept that everyone has the
> right to question everyone else in the organization.  In a company of over
> 400,000 employees spread around the world adherence to this concept might
> easily breed chaos.  While we might accept that a tester might have some
> good ideas that an MBA in marketing with 20 years experience might not have,
> it's hard to believe that someone in marketing has any idea of what testing
> is about much less provide any valuable feedback especially when the testing
> group and marketing group and psychologically as well as geographically
> disparate.
>
> Thanks for the explanation and I now understand what you are saying about
> "cooking".  That was not my intent.  I am only trying to engender the same
> rigor in testers dealing with management that the testers apply to the
> systems they are testing.
>
> Your comments about how it should be does suggest a question though.  I'm
> meeting with some of the senior managers on the business side this week.  My
> approach has been to attempt to move control of testing to a more central
> body, namely the testers under QA.  As said before in my note to Lisa, the
> developers do the unit, integration and system testing and the users do the
> acceptance testing independently of QA.  QA does testing between systems and
> acceptance.  There are the usual fingers pointed as you can imagine and
> lines drawn in the sand and so forth.
> What I'm wondering is whether you might have a suggestion of how to present
> a pitch for more agile approaches considering that the bulk of their
> processes - in the tens of millions of transactions a day, operating 24x7 -
> are on backend mainframes with code written in 3GL and 4GL.  There are
> "pockets of agile" somewhere, but I haven't stumbled upon them as yet in the
> eight weeks I've been here.  Of course, I've only been dealing with the QA
> department so far.  I'd love to be able to have a stronger agile pitch to
> senior management, but am at a loss.  I'm focusing on repairing the
> communications breakdowns up and down the lines at this moment.  Note that
> in two of the locations I've been to so far there has been a similar
> refrain: the testers and developers should work it out and stop their
> complaining.  Underlying message:  there are plenty out there looking for
> work if you don't like it here.
> Any suggestions?
> Thanks in advance.
> =steve
>
>
>
>  *From:* Steven Gordon <sgordonphd@...>
> *Sent:* Sunday, September 20, 2009 11:59 AM
> *To:* agile-testing@...
> *Subject:* Re: QRe: [agile-testing] "Why is QA Always the bottleneck?"
>
>
>
> Does the financial department have to justify the need for accounting
> standards and independent audits?  (Events over the last decade indicate
> that that case may have been lost at many companies - it is a case that
> should never even be entertained.)
>
> The VP of Marketing can certainly be concerned and give feedback about
> delivery times, but has just as much right to question the role of testing
> as the developers do questioning the specifics of how Marketing decides what
> features are needed, when they are needed and when to announce them (of
> course, Marketing can be given feedback on what it does just not
> micro-managed on their internal processes).
>
> I am not questioning your cost model that tries to explain the relationship
> between testing and risk, just stating that the actual numbers you feed into
> that model are pure guesses and can easily be manipulated to get whatever
> result you want.  If you present it as:
> - if the numbers were A1, A2, ... An, the model would predict A,
> - if the numbers were instead B1, B2, .. Bn, the model would instead
> predict B,
> - etc.
> in order to depict the relationship between testing and risk, I would not
> call it cooking.  But, if you provide just one set of inputs as if they are
> known values, then you are cooking the numbers.
>
> This case is between development and the VP of Development.  If the VP of
> development buys it, he has to say to upper management only that his
> department will take responsibility to deliver as quickly as they
> responsibly can.  If the VP of development does not buy it, then you cannot
> do agile software development or maybe even professionally responsible
> software development.
>
> On Sun, Sep 20, 2009 at 6:49 AM, Steve Blais <sialbs@...> wrote:
>
>>
>>
>> Not cooking the numbers, Steve.  Simply taking the time to identify and
>> record the numbers that are there inherently in the process rather than
>> simply going on faith that management will see the light and realize the
>> importance of testing.  Most likely management will come to that
>> realization, but only after some major disaster occurs for which the testers
>> will be blamed.
>>  The thing to keep in mind is that while we understand the professional
>> approach to software development the VP of Marketing, who is the heir
>> apparent to the CEO and has the Board's ear, does not. The VP Marketing is
>> only concerned about the precipitous drop in sales that he perceives comes
>> from slow delivery of new products from IT.
>>
>>
>>  *From:* Steven Gordon <sgordonphd@...>
>> *Sent:* Saturday, September 19, 2009 12:47 PM
>>  *To:* agile-testing@...
>> *Subject:* Re: QRe: [agile-testing] "Why is QA Always the bottleneck?"
>>
>>
>>
>> On Sat, Sep 19, 2009 at 2:07 AM, Steve Blais <sialbs@...> wrote:
>>
>> >
>> >
>> >
>> > Can you suggest a way of proving to Upper Management the value of
>> testing other than with risk reduction?  This would help us convince the PTB
>> (powers that be) that it would be cost-effective to move QA back in the SDLC
>> and engage more with the developers (developers attitudes aside).
>> >
>>
>> Just integrate QA (which is more than just testing), as well as every
>> other software development specialization, into software development
>> so seamlessly that upper management just sees a professional software
>> development department, not the individual pieces to be micromanaged.
>>
>> If upper management does not allow this, then you have to find a way
>> to tell the following to upper management (in more polite terms):
>>
>> We are the professional software developers. We take responsibility
>> to develop software in a professional manner. Do you ask all the
>> other departments to justify their professional practices for
>> accomplishing their goals and responsibilities?
>>
>> I, like George, would feel unprofessional cooking the numbers to
>> justify what you know is the professional way to develop software.
>>
>>  ------------------------------
>>
>>
>> No virus found in this incoming message.
>> Checked by AVG - www.avg.com
>> Version: 8.5.409 / Virus Database: 270.13.108/2383 - Release Date:
>> 09/19/09 17:50:00
>>
>
>  ------------------------------
>
>
> No virus found in this incoming message.
> Checked by AVG - www.avg.com
> Version: 8.5.409 / Virus Database: 270.13.109/2384 - Release Date: 09/20/09
> 06:22:00
>  
>

[Steven Gordon-2]

My first sentence is missing a 'not'.

On Mon, Sep 21, 2009 at 8:50 AM, Steven Gordon <sgordonphd@...> wrote:

> It seems like this organization has not adopted agile top-down, so my
> previous comments are fully applicable to such an organization,
>
> Top-down agile adoption cannot be successful in an organization that
> believes it is doing just fine with its current approach.  The attitude that
> there are lots of people waiting to fill the jobs of people who do not like
> the way things are done here indicates that the organization as a whole is
> far from ready to consider a different approach.
>
> Given that, agile approaches can still work in isolated sub-organizations,
> but the people who interface between an agile sub-organization and the rest
> of the organization face the formidable challenge of living in both worlds.
>
> Centralizing QA is a step away from agility. It may solve the most visible
> problems today, but I believe it will always create worse problems in the
> long run.  It adds extra communication overhead, increases the size of
> feedback loops, and breaks down the responsibility for delivering
> high-quality working software in a way such that nobody is really
> responsible.
>
> The cetnral argument at a non-agile organization for integrating QA into
> each project team is that doing so now makes the team itself responsible for
> deliveribng software.  This does not necessarily mean that the QA people on
> the project team might not have also report to a centralized QA authority,
> especially in an already matrixed organization, but the daily responsibility
> is to collaborate with everybody else on the project team to deliver working
> software early and often.  Again, this also includes other specializations
> such as DBAs, BAs, documentors and trainers (although some may not have
> enough work to be full-time members of the project team during the entire
> life cycle).
>
> Once the product owner(s) and development team agree to this, then the next
> step is to move from waterfall-like approach to a leaner user-story-based
> approach that allows delivery of working software every two weeks without
> entire use cases being fully implemented in every delivery.  The deliveries
> do not have to be deployable, but the rhythm of delivering software that
> works and getting concrete feedback on that delivery convinces the business
> people that the team is actually delivering, not twiddling their thumbs
> creating designs on paper until just before the release date.  It also means
> there will always be something to deliver on release date, even if a few
> features have slipped, as opposed to the all-or-nothing delivery the company
> may well be used to.
>
> SteveG
>
> On Mon, Sep 21, 2009 at 4:46 AM, Steve Blais <sialbs@...> wrote:
>
>>
>>
>> Hi Steve
>> I understand and am in agreement with the concept that everyone has the
>> right to question everyone else in the organization.  In a company of over
>> 400,000 employees spread around the world adherence to this concept might
>> easily breed chaos.  While we might accept that a tester might have some
>> good ideas that an MBA in marketing with 20 years experience might not have,
>> it's hard to believe that someone in marketing has any idea of what testing
>> is about much less provide any valuable feedback especially when the testing
>> group and marketing group and psychologically as well as geographically
>> disparate.
>>
>> Thanks for the explanation and I now understand what you are saying about
>> "cooking".  That was not my intent.  I am only trying to engender the same
>> rigor in testers dealing with management that the testers apply to the
>> systems they are testing.
>>
>> Your comments about how it should be does suggest a question though.  I'm
>> meeting with some of the senior managers on the business side this week.  My
>> approach has been to attempt to move control of testing to a more central
>> body, namely the testers under QA.  As said before in my note to Lisa, the
>> developers do the unit, integration and system testing and the users do the
>> acceptance testing independently of QA.  QA does testing between systems and
>> acceptance.  There are the usual fingers pointed as you can imagine and
>> lines drawn in the sand and so forth.
>> What I'm wondering is whether you might have a suggestion of how to
>> present a pitch for more agile approaches considering that the bulk of their
>> processes - in the tens of millions of transactions a day, operating 24x7 -
>> are on backend mainframes with code written in 3GL and 4GL.  There are
>> "pockets of agile" somewhere, but I haven't stumbled upon them as yet in the
>> eight weeks I've been here.  Of course, I've only been dealing with the QA
>> department so far.  I'd love to be able to have a stronger agile pitch to
>> senior management, but am at a loss.  I'm focusing on repairing the
>> communications breakdowns up and down the lines at this moment.  Note that
>> in two of the locations I've been to so far there has been a similar
>> refrain: the testers and developers should work it out and stop their
>> complaining.  Underlying message:  there are plenty out there looking for
>> work if you don't like it here.
>> Any suggestions?
>> Thanks in advance.
>> =steve
>>
>>
>>
>>  *From:* Steven Gordon <sgordonphd@...>
>> *Sent:* Sunday, September 20, 2009 11:59 AM
>> *To:* agile-testing@...
>> *Subject:* Re: QRe: [agile-testing] "Why is QA Always the bottleneck?"
>>
>>
>>
>> Does the financial department have to justify the need for accounting
>> standards and independent audits?  (Events over the last decade indicate
>> that that case may have been lost at many companies - it is a case that
>> should never even be entertained.)
>>
>> The VP of Marketing can certainly be concerned and give feedback about
>> delivery times, but has just as much right to question the role of testing
>> as the developers do questioning the specifics of how Marketing decides what
>> features are needed, when they are needed and when to announce them (of
>> course, Marketing can be given feedback on what it does just not
>> micro-managed on their internal processes).
>>
>> I am not questioning your cost model that tries to explain the
>> relationship between testing and risk, just stating that the actual numbers
>> you feed into that model are pure guesses and can easily be manipulated to
>> get whatever result you want.  If you present it as:
>> - if the numbers were A1, A2, ... An, the model would predict A,
>> - if the numbers were instead B1, B2, .. Bn, the model would instead
>> predict B,
>> - etc.
>> in order to depict the relationship between testing and risk, I would not
>> call it cooking.  But, if you provide just one set of inputs as if they are
>> known values, then you are cooking the numbers.
>>
>> This case is between development and the VP of Development.  If the VP of
>> development buys it, he has to say to upper management only that his
>> department will take responsibility to deliver as quickly as they
>> responsibly can.  If the VP of development does not buy it, then you cannot
>> do agile software development or maybe even professionally responsible
>> software development.
>>
>> On Sun, Sep 20, 2009 at 6:49 AM, Steve Blais <sialbs@...> wrote:
>>
>>>
>>>
>>> Not cooking the numbers, Steve.  Simply taking the time to identify and
>>> record the numbers that are there inherently in the process rather than
>>> simply going on faith that management will see the light and realize the
>>> importance of testing.  Most likely management will come to that
>>> realization, but only after some major disaster occurs for which the testers
>>> will be blamed.
>>>  The thing to keep in mind is that while we understand the professional
>>> approach to software development the VP of Marketing, who is the heir
>>> apparent to the CEO and has the Board's ear, does not. The VP Marketing is
>>> only concerned about the precipitous drop in sales that he perceives comes
>>> from slow delivery of new products from IT.
>>>
>>>
>>>  *From:* Steven Gordon <sgordonphd@...>
>>> *Sent:* Saturday, September 19, 2009 12:47 PM
>>>  *To:* agile-testing@...
>>> *Subject:* Re: QRe: [agile-testing] "Why is QA Always the bottleneck?"
>>>
>>>
>>>
>>> On Sat, Sep 19, 2009 at 2:07 AM, Steve Blais <sialbs@...> wrote:
>>>
>>> >
>>> >
>>> >
>>> > Can you suggest a way of proving to Upper Management the value of
>>> testing other than with risk reduction?  This would help us convince the PTB
>>> (powers that be) that it would be cost-effective to move QA back in the SDLC
>>> and engage more with the developers (developers attitudes aside).
>>> >
>>>
>>> Just integrate QA (which is more than just testing), as well as every
>>> other software development specialization, into software development
>>> so seamlessly that upper management just sees a professional software
>>> development department, not the individual pieces to be micromanaged.
>>>
>>> If upper management does not allow this, then you have to find a way
>>> to tell the following to upper management (in more polite terms):
>>>
>>> We are the professional software developers. We take responsibility
>>> to develop software in a professional manner. Do you ask all the
>>> other departments to justify their professional practices for
>>> accomplishing their goals and responsibilities?
>>>
>>> I, like George, would feel unprofessional cooking the numbers to
>>> justify what you know is the professional way to develop software.
>>>
>>>  ------------------------------
>>>
>>>
>>> No virus found in this incoming message.
>>> Checked by AVG - www.avg.com
>>> Version: 8.5.409 / Virus Database: 270.13.108/2383 - Release Date:
>>> 09/19/09 17:50:00
>>>
>>
>>  ------------------------------
>>
>>
>> No virus found in this incoming message.
>> Checked by AVG - www.avg.com
>> Version: 8.5.409 / Virus Database: 270.13.109/2384 - Release Date:
>> 09/20/09 06:22:00
>>  
>>
>
>

[Bradley, Todd]

All this talk about cost justifications, etc. is strange on the
"agile-testing" list.

 

Whenever my boss asks me a question like "why is [testing] always the
bottleneck" I have a quick and easy answer.

 

"I think I speak for the whole team when I say we'd be glad to spend the
next month on a tropical beach if that's what you'd prefer."

 

My current boss is smart enough to know the process is better off with
testers than without.  If I should ever happen to work again for one
that isn't that smart, then he'll get rid of his "bottleneck" and I'll
get a suntan and we'll both be happy!  J

 

 

Cheers,

Todd.

 

 

From: agile-testing@...
[mailto:agile-testing@...] On Behalf Of Steve Blais
Sent: Friday, September 18, 2009 4:26 PM
To: agile-testing@...
Subject: Re: QRe: [agile-testing] "Why is QA Always the bottleneck?"

 






OK.

 

Assumption:  when management (or anyone) accuses the test team of being
a bottleneck, what they are saying is "you are not providing a value for
the time you are taking that justifies the time you are taking,
therefore you are unjustifiably holding up progress - a bottleneck."
When the test team is able to show that it is providing value then it
proves that it is not a bottleneck.  Therefore...

 

Step 1:  determine the value of the product to the organization.

   Let's use a simple example of making modifications or enhancements to
a web site.  Checking with sales administration we determine that the
current website averages $240K in revenue a day. (actual figures are
higher, but we'll round it to an easy number for the example).

 

Step 2: identify an understandable and quantifiable risk to the value.

   We'll use the obvious one: web site crashes.  We can check our
availability calculations for this. MTBF does not really help, but MTTR
gives us the information we need.  The MTTR is 42.28 so for the example
we'll make it 30 minutes.  Therefore the quantifiable risk is $50K.

 

Step 3:  compute the cost of the testing

   Assume 3 testers on the team doing nothing but working on the System
Under Test.  For argument's sake, their combined gross pay is 3000
(average gross pay of 52K) which means the fully loaded cost runs out
about 6200/week.  

 

Now the challenge is to link the cost of the testing (disregarding the
overhead costs of equipment, space, etc.) to the reduction of the risk.

 

A note about risk reduction.  Risk, by definition, is uncertainty.
Testing is about removing the uncertainty associated with any software
product. When the product is introduced to the testing group there is a
100% chance of defects in the product  or 100% uncertainty about the
product's success. The job of testing is to remove that uncertainty.
When a test is executed (any form of test) it removes uncertainty
whether the result is that no defect was found, the product works but is
defective, or the product fails as a result of the test.  In any case,
there is no further uncertainty and therefore no risk.  When we find a
defect we no longer have a risk that the defect exists, we have a fact:
it is there. Once a defect has been found, someone else - typically
someone in management or authority - determines whether to fix it or
not. Either way, the risk is eliminated.

 

Approach 1 - saturation

At any given point in time (the point at which the bottleneck accusation
takes place for example) there is a finite set of requirements (use
cases, user stories, items on the burn down list, etc.) which defines
the product identified in Step 1 above.  Assuming change will take
place, the discussion is about what we know now, and we cannot
anticipate what change might take place. Given a finite set of product
describers, an equally finite set of tests can be devised that will
exercise the product at this point in time.

For the example, we'll make two assumptions, both of which are false in
reality, but they help make the math easier:

1. each test of the product is equal in terms of satisfying product
describers and uncertainty reduction, and
2. defects are equally distributed among all product describers

 

At the bottleneck discussion 50 % of the tests have been executed
reducing the remaining risk to 50%.  The time to execute the remaining
tests is independent of the time it took to execute the first 50%, but
let's arbitrarily say that it will take another two weeks to complete
the tests to remove all uncertainty in the product.  Assuming that in
the first 50% of tests the testing team found 5 severe defects (web
crashers), then mathematically at this point the organization faces a
risk that there are another 5 remaining and the quantifiable risk of
that is $250K. Since the cost to identify those risks (and ostensibly
get them fixed) is $12,400, there is no bottleneck; it makes good
business sense to continue to test.

 

Keep in mind that when the defects are fixed the uncertainty goes back
to 100% until we prove through testing that the defects have been
removed.  

 

The approach is simplified for example's sake, but the approach is still
valid.

 

OR

Approach 2 : product risk assessment

 

Someone, typically the business analyst or quality assurance, should
have performed a product risk analysis.  This is different than the
project risk analysis that the project manager performs.  Based on the
analysis we have a vulnerability value.  We also are able to prioritize
our quality assurance activities based on the identified risks to the
organization due to failure of the product and remove the risk
(uncertainty) through testing the high risk items.  We also create a
measuring stick for the bottleneck discussion.  Based on the assigned
value of the risk items compared to the overall vulnerability we can
easily determine the percentage of risk remaining. Again, we can project
the number of tests or parts of the product that need to be exercised to
eliminate the specified uncertainty and, based on percent complete,
reasonably state that the specified risk has been reduced by that
percentage.  There are mathematical algorithms that can assign a
quantifiable dollar value to the impact value of the risk assessment so
that you can use dollars to demonstrate the value of continued testing.
In the aggregate, this approach is weak, but probably will hold up in
bottleneck discussions with management, especially if you take the
precision to three decimal points.  

 

OR

Approach 3: problem solving

 

At the beginning of the project or initiative or whatever, someone,
typically the business analyst gets answers to the following:

Is this the problem we need to solve? (statement of problem)

When "yes", what do you see as the result of providing a solution?
(Statement of vision)

What do you need to see to prove that we have solved your problem?
(acceptance criteria)

 

Based on the answer to the third question, a set of tests can be created
to provide the proof requested.  Assuming that we start with a 100% risk
that the product will not meet these tests, we can compute a percentage
based on the number of those tests executed, each reducing the
uncertainty. As we test, the percentage of uncertainty or risk is
reduced.

 

However, there is the other side of the coin.  Suppose that the three
testers are in end game testing when the accusation is leveled at them.
They are in retesting of bug-fixes and performing more extensive tests
of the software.  The statistics show that the defect discovery rate is
now at four defects per day and they have not unearthed a high severity
defect in several weeks. This means that the team's position that more
testing is necessary, assuming the system has been exercised completely
at least once, becomes less defensible.  Management's return for the
$6200 for an extra week's testing does not include any decreased risk to
the effectiveness of the product. In this case, testing might well be
considered a bottleneck: the cost of testing is greater than the value
that the testing returns to the organization.

 

Further complications arise when marketing, which as we know never lies,
predicts that the organization stands to gain $50K worth of new revenue
for every day that the web site is operational.  Now the quantifiable
impact increases significantly, but so does the impact of testing one
more day to find that Elusive Defect that Justifies Your Existence
(EDJYE).  The bottleneck designation may well be foisted on the testers
unless evidence or algorithms can be proffered to show risk that offsets
that marketing-guaranteed $50K per day. Or we can make a good argument
for a threat to intangible benefits such as reputation.

 

 

 

 

From: George Dinwiddie <mailto:lists@...>  

Sent: Thursday, September 17, 2009 9:29 AM

To: agile-testing@...

Subject: Re: QRe: [agile-testing] "Why is QA Always the bottleneck?"

 

 

Steve Blais wrote:
> What management needs is some facts - that is, some dollars and sense
> justification. "In three weeks of testing we have reduced the risk of
> releasing this product from 100% to 40%. At 40% there is a likelihood
> that the product will fail for some reason once a day and be done an
> average of 20 minutes. Marketing estimates a loss of $X for every
> failure. With another week of testing we can reduce the risk to 20%
> resulting in only $X risk. What would you like to do?"

How do you calculate these numbers?

- George

--
----------------------------------------------------------
* George Dinwiddie * http://blog.gdinwiddie.com
Software Development http://www.idiacomputing.com
Consultant and Coach http://www.agilemaryland.org
----------------------------------------------------------

________________________________


No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 8.5.409 / Virus Database: 270.13.102/2377 - Release Date:
09/16/09 17:49:00








 

[R. Sankara Narayanan-2]

--------------------------------------------------
From: "Bradley, Todd" <todd.bradley@...>
Sent: Tuesday, September 22, 2009 3:52 PM

> My current boss is smart enough to know the process is better off with
> testers than without.  If I should ever happen to work again for one
> that isn't that smart, then he'll get rid of his "bottleneck" and I'll
> get a suntan and we'll both be happy!  J
>

Why should we allow the boss to get rid of the testers which is not good
for the project anyways? Why not the testers try to get rid of the
ignorant boss and bring success to the project?

Cheers,
Sanky,
http://www.sodhanai.com
sodhanai@...

[sialbs]

I believe you are right on the money which is an unsubtle way of saying our thoughts on the subject match.  My approach in "centralizing" the QA function was to get someone to have a view of the entire SDLC and my thinking (especially since I'm here at the behest of QA) is that this can be most successfully accomplished under the guise of quality product delivery, as opposed to, say, creating a PMO or some other artificial oversight governance body.  My long term goal is to have the senior QA people become more process oriented than test-oriented so that we can have an internal body that can start the long, slow move toward agility.

But talk about irony.

Near the end of the morning meeting yesterday a director from the business side, one with significantly more IT savvy than the others announces that the organization is making a firm commitment to becoming more agile.  That got my attention.  Except that what was meant by "more agile" is this:  they currently operate on a six-month release cycle with two major releases a year, January and July.  Right now the testers are testing the Jan 10 release, the developers are working on the July 10 release and fixes to the Jan 10 release and the business analysts are completing the requirements for the July 10 release and starting the preparation of the requirements for the Jan 11 release. (As an example of the total separation of functionality, the business analyst drops all relationship with a product release once the requirements are approved in a series of meetings and then goes on to the next release or perhaps a different product taking all knowledge with him.  One of my recommendations is to have the business analyst stay with the product from requirements definition through to product release to have at least one person responsible for the product from beginning to end.  This is my way of sneaking in the role of product manager.) What was meant by "more agile" is that they are "working toward" shorter and more frequent release cycles. The fellow from IT sitting next to the "agile" director mentioned that they were thinking about releases every 90 days to start with and eventually getting down to two - three week cycles which would make them "agile".
This is an indication of the adoption of the rhetoric that many of the agile proponents have laid on the industry of shortened time frames and therefore higher productivity.  The proponents don't mention things like the increased discipline, newly acquired skillsets, structure, and underlying significant requirement for cross-functional collaboration.
Unfortunately, the meeting ended with a lunch call and we were on different subjects with a different cast of characters with whom the agile subject was not approachable, so I could neither pursue how a mainframe-based organization intended to adopt this "agility" or post a few objects or advisories.
Your postulations in these posts about management promoting their concept of "agile" without any commitment to changing culture or organization. (For example, my suggestion a week or so ago about assigning an SME to monitor the progress of the product from the business side (another of my attempts to foster a product owner mentality) instead of talking with the business analyst and then 15 months later coming back to manage the acceptance testing was met with immediate dismissal.  Primary reason:  our main issues are with foreign languages and countries using our products outside the US and we don't have only one person who is able to ensure all the translations are correct and the products adhere to all local regulations.  In other words, they missed the point entirely, but the discussion on the subject was ended.)  Your statements that such organizations are not truly agile, and perhaps that is best for them are also on the money.  I had decided to heed your words and focus on increasing collaboration in the culture to solve some internal process problems instead of trying to "agile-ize" any parts of the organization, and then we have this meeting.
Last night I talked to a couple of other people in the trenches and they confirmed that there is indeed a movement in this location, which has a bit of seniority over the other locations I have been working in, toward this notion of agility by reducing release cycles and letting the troops work out how to get the job done faster.

I do get another shot with these two in a meeting tomorrow which has a different agenda, but I might be able to sneak in a conversation about how they intend to bring in "agile".  I'm interested in seeing how close their response, if I can get one, parallels the thoughts you express below.
thanks for the irony
=steve


From: Steven Gordon
Sent: Monday, September 21, 2009 11:50 AM
To: agile-testing@...
Subject: Re: QRe: [agile-testing] "Why is QA Always the bottleneck?"


  It seems like this organization has not adopted agile top-down, so my previous comments are fully applicable to such an organization,

Top-down agile adoption cannot be successful in an organization that believes it is doing just fine with its current approach.  The attitude that there are lots of people waiting to fill the jobs of people who do not like the way things are done here indicates that the organization as a whole is far from ready to consider a different approach.

Given that, agile approaches can still work in isolated sub-organizations, but the people who interface between an agile sub-organization and the rest of the organization face the formidable challenge of living in both worlds.

Centralizing QA is a step away from agility. It may solve the most visible problems today, but I believe it will always create worse problems in the long run.  It adds extra communication overhead, increases the size of feedback loops, and breaks down the responsibility for delivering high-quality working software in a way such that nobody is really responsible.

The cetnral argument at a non-agile organization for integrating QA into each project team is that doing so now makes the team itself responsible for deliveribng software.  This does not necessarily mean that the QA people on the project team might not have also report to a centralized QA authority, especially in an already matrixed organization, but the daily responsibility is to collaborate with everybody else on the project team to deliver working software early and often.  Again, this also includes other specializations such as DBAs, BAs, documentors and trainers (although some may not have enough work to be full-time members of the project team during the entire life cycle).

Once the product owner(s) and development team agree to this, then the next step is to move from waterfall-like approach to a leaner user-story-based approach that allows delivery of working software every two weeks without entire use cases being fully implemented in every delivery.  The deliveries do not have to be deployable, but the rhythm of delivering software that works and getting concrete feedback on that delivery convinces the business people that the team is actually delivering, not twiddling their thumbs creating designs on paper until just before the release date.  It also means there will always be something to deliver on release date, even if a few features have slipped, as opposed to the all-or-nothing delivery the company may well be used to.

SteveG



On Mon, Sep 21, 2009 at 4:46 AM, Steve Blais <sialbs@...> wrote:

   

  Hi Steve
  I understand and am in agreement with the concept that everyone has the right to question everyone else in the organization.  In a company of over 400,000 employees spread around the world adherence to this concept might easily breed chaos.  While we might accept that a tester might have some good ideas that an MBA in marketing with 20 years experience might not have, it's hard to believe that someone in marketing has any idea of what testing is about much less provide any valuable feedback especially when the testing group and marketing group and psychologically as well as geographically disparate.

  Thanks for the explanation and I now understand what you are saying about "cooking".  That was not my intent.  I am only trying to engender the same rigor in testers dealing with management that the testers apply to the systems they are testing.  

  Your comments about how it should be does suggest a question though.  I'm meeting with some of the senior managers on the business side this week.  My approach has been to attempt to move control of testing to a more central body, namely the testers under QA.  As said before in my note to Lisa, the developers do the unit, integration and system testing and the users do the acceptance testing independently of QA.  QA does testing between systems and acceptance.  There are the usual fingers pointed as you can imagine and lines drawn in the sand and so forth.  
  What I'm wondering is whether you might have a suggestion of how to present a pitch for more agile approaches considering that the bulk of their processes - in the tens of millions of transactions a day, operating 24x7 - are on backend mainframes with code written in 3GL and 4GL.  There are "pockets of agile" somewhere, but I haven't stumbled upon them as yet in the eight weeks I've been here.  Of course, I've only been dealing with the QA department so far.  I'd love to be able to have a stronger agile pitch to senior management, but am at a loss.  I'm focusing on repairing the communications breakdowns up and down the lines at this moment.  Note that in two of the locations I've been to so far there has been a similar refrain: the testers and developers should work it out and stop their complaining.  Underlying message:  there are plenty out there looking for work if you don't like it here.  
  Any suggestions?
  Thanks in advance.
  =steve




  From: Steven Gordon
  Sent: Sunday, September 20, 2009 11:59 AM
  To: agile-testing@...
  Subject: Re: QRe: [agile-testing] "Why is QA Always the bottleneck?"


   
  Does the financial department have to justify the need for accounting standards and independent audits?  (Events over the last decade indicate that that case may have been lost at many companies - it is a case that should never even be entertained.)

  The VP of Marketing can certainly be concerned and give feedback about delivery times, but has just as much right to question the role of testing as the developers do questioning the specifics of how Marketing decides what features are needed, when they are needed and when to announce them (of course, Marketing can be given feedback on what it does just not micro-managed on their internal processes).

  I am not questioning your cost model that tries to explain the relationship between testing and risk, just stating that the actual numbers you feed into that model are pure guesses and can easily be manipulated to get whatever result you want.  If you present it as:
  - if the numbers were A1, A2, ... An, the model would predict A,
  - if the numbers were instead B1, B2, .. Bn, the model would instead predict B,
  - etc.
  in order to depict the relationship between testing and risk, I would not call it cooking.  But, if you provide just one set of inputs as if they are known values, then you are cooking the numbers.

  This case is between development and the VP of Development.  If the VP of development buys it, he has to say to upper management only that his department will take responsibility to deliver as quickly as they responsibly can.  If the VP of development does not buy it, then you cannot do agile software development or maybe even professionally responsible software development.  



  On Sun, Sep 20, 2009 at 6:49 AM, Steve Blais <sialbs@...> wrote:

     

    Not cooking the numbers, Steve.  Simply taking the time to identify and record the numbers that are there inherently in the process rather than simply going on faith that management will see the light and realize the importance of testing.  Most likely management will come to that realization, but only after some major disaster occurs for which the testers will be blamed.
     The thing to keep in mind is that while we understand the professional approach to software development the VP of Marketing, who is the heir apparent to the CEO and has the Board's ear, does not. The VP Marketing is only concerned about the precipitous drop in sales that he perceives comes from slow delivery of new products from IT.



    From: Steven Gordon
    Sent: Saturday, September 19, 2009 12:47 PM
    To: agile-testing@...
    Subject: Re: QRe: [agile-testing] "Why is QA Always the bottleneck?"


     
    On Sat, Sep 19, 2009 at 2:07 AM, Steve Blais <sialbs@...> wrote:


    >
    >
    >
    > Can you suggest a way of proving to Upper Management the value of testing other than with risk reduction?  This would help us convince the PTB (powers that be) that it would be cost-effective to move QA back in the SDLC and engage more with the developers (developers attitudes aside).
    >

    Just integrate QA (which is more than just testing), as well as every
    other software development specialization, into software development
    so seamlessly that upper management just sees a professional software
    development department, not the individual pieces to be micromanaged.

    If upper management does not allow this, then you have to find a way
    to tell the following to upper management (in more polite terms):

    We are the professional software developers. We take responsibility
    to develop software in a professional manner. Do you ask all the
    other departments to justify their professional practices for
    accomplishing their goals and responsibilities?

    I, like George, would feel unprofessional cooking the numbers to
    justify what you know is the professional way to develop software.



----------------------------------------------------------------------------



    No virus found in this incoming message.
    Checked by AVG - www.avg.com

    Version: 8.5.409 / Virus Database: 270.13.108/2383 - Release Date: 09/19/09 17:50:00






------------------------------------------------------------------------------



  No virus found in this incoming message.
  Checked by AVG - www.avg.com

  Version: 8.5.409 / Virus Database: 270.13.109/2384 - Release Date: 09/20/09 06:22:00







--------------------------------------------------------------------------------



No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 8.5.409 / Virus Database: 270.13.109/2384 - Release Date: 09/20/09 06:22:00