QA As UX On Agile Projects?

Hello. My name is Doug Gorman. I'm the QA manager at Creative Artists.

Recently we've begun to take on usability testing as a part of our overall project testing. Have any of you been on or led QA teams into a usability focus? If so, I'd love to benefit from your experience.

What worked or didn't work in switching your team's mind set to incorporate usability?

What resources did you use to teach/learn usability?

What usability testing practices did you find gave you a lot of value in terms of identifying interface issues earlier?

What big mistakes did you learn from early on?

Looking back, are there "pre-requisites" to establishing a usability focus that you wish you had implemented earlier?

[chrs_mcmhn]

--- In agile-testing@..., "doug.gorman@..." <doug.gorman@...> wrote:
>
> What worked or didn't work in switching your team's mind set to incorporate usability?
>
> What resources did you use to teach/learn usability?

I'm only an amateur at this kind of work, but I follow it some.  I'd recommend you read anything you can find by Jeff Patton.  He is just about the best in the business at this sort of thing.

[kerrykimbrough]

Hi, Doug

You're in for an interesting ride. There is a lot of synergy between UI dev and agility. In fact, UI designers have been preaching and following agile principles since before the Manifesto.

One issue I'm sure you've already begun to address is the relationship between your UX test team and the UX dev team. Since the UX dev team is aiming for high usability, they probably have worked out some premises and principles to guide their design. Your team needs to learn and adopt them, too. The principle which says that "X is usable" is also the oracle that says that "Y's usability is flawed". Ideally, your teams can collaborate early to shape this oracle. You can help the UX dev team make explicit what is often tacit, private, and potentially inconsistent. Also, such design rules benefit from a testing perspective.

> What worked or didn't work in switching your team's mind set to incorporate usability?

In my experience, the biggest hurdle is getting the team on board with a clear consensus on what a usability defect is. The focus is not on the system. Instead, it's on the user's work tasks. A usability defect is any detriment to user A completing task B to satisfaction. A UX tester *must* know and understand user A and his work. It's a challenge to acquire this knowledge and it's out of the comfort zone for some. But without it, you can be trapped in counter-effective "opinion wars", with the dev team and even within the test team. On the other hand, this knowledge is important for almost any test team, so it shouldn't be too much of a stretch.

> What resources did you use to teach/learn usability?

There are lots, but I especially recommend the work of Jakob Nielsen. He is thorough, pragmatic, insightful, and engineering-oriented. His book "Usability Engineering" is very good. Also, the materials on his Web site, www.useit.com. You may find his ideas on "discount usability testing" especially interesting.

> What usability testing practices did you find gave you a lot of value in terms of identifying interface issues earlier?

One practice I like is the "duo test". It's cheap and effective and doesn't require super expertise. Here's how it works. A team of 2 testers conducts the exercise, in which user subjects use the system to perform a specific task and testers observe what happens, looking for usability issues. There are also 2 user subjects, who work together as a team to complete the tasks. This is crucial, because it encourages users to think out loud and discuss problems in ways that thus become more observable. One tester is the scribe and just takes notes about observations. The other tester is the guide who moderates the exercise, answers questions, highlights key observations, and generally keeps things rolling. This is a form of the sort of "heuristic evaluation" that Nielsen describes. You can do a "duo test" with a paper storyboard, a mockup, or an actual system.

> What big mistakes did you learn from early on?

The first time I did a duo test, I tried to video tape it. It was awful and useless. An audio-only recording would have been more useful, but I now think that good notes are best.

Regards,
Kerry

[Coyle, Jim]

Hi,

if anyone has any experience of fuzz testing (aka fuzzing), could they please share?
I'm interested in the good and the bad.

thanks
Jim



If you've received this email by mistake, we're sorry for bothering you. It may contain information that's confidential, so please delete it without sharing it. And if you let us know, we can try to stop it from happening again. Thank you.

We may monitor any emails sent or received by us, or on our behalf. If we do, this will be in line with relevant law and our own policies.

Sage Ireland, trading name of Sage Hibernia Limited. Registered in Ireland at 3096 Lake Drive, City West Business Park, Dublin 24. Registered number 300549.

Thanks for the advice! I signed up for Jeff's usability group here on Yahoo - agile-usability - and posted a similar question there.

--- In agile-testing@..., "chrs_mcmhn" <christopher.mcmahon@...> wrote:
>
> --- In agile-testing@..., "doug.gorman@" <doug.gorman@> wrote:
> >
> > What worked or didn't work in switching your team's mind set to incorporate usability?
> >
> > What resources did you use to teach/learn usability?
>
> I'm only an amateur at this kind of work, but I follow it some.  I'd recommend you read anything you can find by Jeff Patton.  He is just about the best in the business at this sort of thing.
>

[Penny Wyatt]

Hi Jim,

I was a member of a test team responsible for testing a text engine written in C++ that runs on Windows. Fuzzing was one of the techniques we used for our security testing.

The software had two major input methods: calls to its API, and a number of different file formats it could load. We approached these separately.

For fuzzing the API calls, we used a tool that was internal to our company. It enabled us to define all the possible interfaces to the software and it would randomly iterate through them. If it found anything interesting, such as an assertion failure or a crash, it would concentrate on that area and try various similar cases. After a short period of time, we found we were getting rapidly diminishing returns from it, as it would always end up travelling down the same code paths. To fix this, one of the other testers modified it to link it with a code-coverage module, forcing it to focus on calls that triggered execution of code that it had not yet covered.

This was of great benefit to us and a huge number of bugs were raised and fixed from this process. On the down side, the time to develop the tools was substantial. Also, this approach worked for us because the developers had applied defence-in-depth to their coding: in debug mode, almost every function call was protected by an assertion. If the fuzzer managed to trigger an unexpected state, it would be noticed almost immediately. Without these assertions, we'd probably have got much less out of the API fuzzing process.

We also had automated our checking of memory allocation to ensure that every object allocated was deleted - giving the fuzzer another heuristic to work with.

Fuzzing of input files was fairly similar. At first, we started off with a tool that randomly swapped bytes in "good" template files, and also generated completely corrupt files. We quickly found that this tended to generate files that were too corrupt to produce interesting test cases - especially for structured file formats like HTML where it doesn't take much to render a file unusable. So we worked from a higher level, making the fuzzer aware of elements within a file format (such as tags or blobs) and allowing it to rearrange them and populate data within them. We found that working with almost-valid data gave us the best results.

The file fuzzing process turned up a few bugs but was not a huge return on investment. The best file input security bugs were still found through a manual process - looking at the code for potential threats, and then manually modifying the files with a hex editor to try to hit those code paths with malicious data.

So, in general; random fuzzing was easy to do but produced minimal benefit for us. Structured, directed fuzzing took more time to set up but produced better results.

It helped to be working with an excellent development team that recognised that being able to get the software into an unexpected state was, of itself, a bug that needed fixing. I have worked in companies where every bug has to be justified with user scenarios before a dev will bother with it; in those places, fuzzing is of minimal benefit because the testers will spend far more time arguing for the need for fixes rather than finding the bugs in the first place.

Hope this helps.

Cheers,
Penny

> Coyle, Jim <jim.coyle@...> wrote:
>
> Hi,
>
> if anyone has any experience of fuzz testing (aka fuzzing), could they
> please share?
> I'm interested in the good and the bad.
>
> thanks
> Jim
>
>
>
> If you've received this email by mistake, we're sorry for bothering you.
> It may contain information that's confidential, so please delete it
> without sharing it. And if you let us know, we can try to stop it from
> happening again. Thank you.
>
> We may monitor any emails sent or received by us, or on our behalf. If
> we do, this will be in line with relevant law and our own policies.
>
> Sage Ireland, trading name of Sage Hibernia Limited. Registered in
> Ireland at 3096 Lake Drive, City West Business Park, Dublin 24.
> Registered number 300549.

[Penny Wyatt]

(Resending because Yahoo appears to have eaten my first reply. My apologies if you receive this twice)


Hi Jim,

I was a member of a test team responsible for testing a text engine written in C++ that runs on Windows. Fuzzing was one of the techniques we used for our security testing.

The software had two major input methods: calls to its API, and a number of different file formats it could load. We approached these separately.

For fuzzing the API calls, we used a tool that was internal to our company. It enabled us to define all the possible interfaces to the software and it would randomly iterate through them. If it found anything interesting, such as an assertion failure or a crash, it would concentrate on that area and try various similar cases. After a short period of time, we found we were getting rapidly diminishing returns from it, as it would always end up travelling down the same code paths. To fix this, one of the other testers modified it to link it with a code-coverage module, forcing it to focus on calls that triggered execution of code that it had not yet covered.

This was of great benefit to us and a huge number of bugs were raised and fixed from this process. On the down side, the time to develop the tools was substantial. Also, this approach worked for us because the developers had applied defence-in-depth to their coding: in debug mode, almost every function call was protected by an assertion. If the fuzzer managed to trigger an unexpected state, it would be noticed almost immediately. Without these assertions, we'd probably have got much less out of the API fuzzing process.

We also had automated our checking of memory allocation to ensure that every object allocated was deleted - giving the fuzzer another heuristic to work with.

Fuzzing of input files was fairly similar. At first, we started off with a tool that randomly swapped bytes in "good" template files, and also generated completely corrupt files. We quickly found that this tended to generate files that were too corrupt to produce interesting test cases - especially for structured file formats like HTML where it
doesn't take much to render a file unusable. So we worked from a higher level, making the fuzzer aware of elements within a file format (such as tags or blobs) and allowing it to rearrange them and populate data within them. We found that working with almost-valid data gave us the best results.

The file fuzzing process turned up a few bugs but was not a huge return on investment. The best file input security bugs were still found through a manual process - looking at the code for potential threats, and then manually modifying the files with a hex editor to try to hit those code paths with malicious data.

So, in general; random fuzzing was easy to do but produced minimal benefit for us. Structured, directed fuzzing took more time to set up but produced better results.

It helped to be working with an excellent development team that recognised that being able to get the software into an unexpected state was, of itself, a bug that needed fixing. I have worked in companies where every bug has to be justified with user scenarios before a dev will bother with it; in those places, fuzzing is of minimal benefit because the testers will spend far more time arguing for the need for fixes rather than finding the bugs in the first place.

Hope this helps.

Cheers,
Penny Wyatt


--- In agile-testing@..., "Coyle, Jim" <jim.coyle@...> wrote:

>
> Hi,
>
> if anyone has any experience of fuzz testing (aka fuzzing), could they please share?
> I'm interested in the good and the bad.
>
> thanks
> Jim
>
>
>
> If you've received this email by mistake, we're sorry for bothering you. It may contain information that's confidential, so please delete it without sharing it. And if you let us know, we can try to stop it from happening again. Thank you.
>
> We may monitor any emails sent or received by us, or on our behalf. If we do, this will be in line with relevant law and our own policies.
>
> Sage Ireland, trading name of Sage Hibernia Limited. Registered in Ireland at 3096 Lake Drive, City West Business Park, Dublin 24. Registered number 300549.
>

[Coyle, Jim]

One thing we do is log usability issues as a category of one of nielsen's 10
heuristics ... http://www.useit.com/papers/heuristic/heuristic_list.html
This keeps it objective (as usability can be subjective)



Thanks for the advice! I signed up for Jeff's usability group here on Yahoo - agile-usability - and posted a similar question there.

--- In agile-testing@...<mailto:agile-testing%40yahoogroups.com>, "chrs_mcmhn" <christopher.mcmahon@...> wrote:
>
> --- In agile-testing@...<mailto:agile-testing%40yahoogroups.com>, "doug.gorman@" <doug.gorman@> wrote:
> >
> > What worked or didn't work in switching your team's mind set to incorporate usability?
> >
> > What resources did you use to teach/learn usability?
>
> I'm only an amateur at this kind of work, but I follow it some. I'd recommend you read anything you can find by Jeff Patton. He is just about the best in the business at this sort of thing.
>




If you've received this email by mistake, we're sorry for bothering you. It may contain information that's confidential, so please delete it without sharing it. And if you let us know, we can try to stop it from happening again. Thank you.

We may monitor any emails sent or received by us, or on our behalf. If we do, this will be in line with relevant law and our own policies.

Sage Ireland, trading name of Sage Hibernia Limited. Registered in Ireland at 3096 Lake Drive, City West Business Park, Dublin 24. Registered number 300549.

[Coyle, Jim]

thanks Penny ...

________________________________
From: agile-testing@... [mailto:agile-testing@...] On Behalf Of Penny Wyatt
Sent: 01 October 2009 22:38
To: agile-testing@...
Subject: Re: [agile-testing] Fuzz testing



Hi Jim,

I was a member of a test team responsible for testing a text engine written in C++ that runs on Windows. Fuzzing was one of the techniques we used for our security testing.

The software had two major input methods: calls to its API, and a number of different file formats it could load. We approached these separately.

For fuzzing the API calls, we used a tool that was internal to our company. It enabled us to define all the possible interfaces to the software and it would randomly iterate through them. If it found anything interesting, such as an assertion failure or a crash, it would concentrate on that area and try various similar cases. After a short period of time, we found we were getting rapidly diminishing returns from it, as it would always end up travelling down the same code paths. To fix this, one of the other testers modified it to link it with a code-coverage module, forcing it to focus on calls that triggered execution of code that it had not yet covered.

This was of great benefit to us and a huge number of bugs were raised and fixed from this process. On the down side, the time to develop the tools was substantial. Also, this approach worked for us because the developers had applied defence-in-depth to their coding: in debug mode, almost every function call was protected by an assertion. If the fuzzer managed to trigger an unexpected state, it would be noticed almost immediately. Without these assertions, we'd probably have got much less out of the API fuzzing process.

We also had automated our checking of memory allocation to ensure that every object allocated was deleted - giving the fuzzer another heuristic to work with.

Fuzzing of input files was fairly similar. At first, we started off with a tool that randomly swapped bytes in "good" template files, and also generated completely corrupt files. We quickly found that this tended to generate files that were too corrupt to produce interesting test cases - especially for structured file formats like HTML where it doesn't take much to render a file unusable. So we worked from a higher level, making the fuzzer aware of elements within a file format (such as tags or blobs) and allowing it to rearrange them and populate data within them. We found that working with almost-valid data gave us the best results.

The file fuzzing process turned up a few bugs but was not a huge return on investment. The best file input security bugs were still found through a manual process - looking at the code for potential threats, and then manually modifying the files with a hex editor to try to hit those code paths with malicious data.

So, in general; random fuzzing was easy to do but produced minimal benefit for us. Structured, directed fuzzing took more time to set up but produced better results.

It helped to be working with an excellent development team that recognised that being able to get the software into an unexpected state was, of itself, a bug that needed fixing. I have worked in companies where every bug has to be justified with user scenarios before a dev will bother with it; in those places, fuzzing is of minimal benefit because the testers will spend far more time arguing for the need for fixes rather than finding the bugs in the first place.

Hope this helps.

Cheers,
Penny

> Coyle, Jim <jim.coyle@...<mailto:jim.coyle%40sage.com>> wrote:
>
> Hi,
>
> if anyone has any experience of fuzz testing (aka fuzzing), could they
> please share?
> I'm interested in the good and the bad.
>
> thanks
> Jim
>
>
>
> If you've received this email by mistake, we're sorry for bothering you.
> It may contain information that's confidential, so please delete it
> without sharing it. And if you let us know, we can try to stop it from
> happening again. Thank you.
>
> We may monitor any emails sent or received by us, or on our behalf. If
> we do, this will be in line with relevant law and our own policies.
>
> Sage Ireland, trading name of Sage Hibernia Limited. Registered in
> Ireland at 3096 Lake Drive, City West Business Park, Dublin 24.
> Registered number 300549.

If you've received this email by mistake, we're sorry for bothering you. It may contain information that's confidential, so please delete it without sharing it. And if you let us know, we can try to stop it from happening again. Thank you.

We may monitor any emails sent or received by us, or on our behalf. If we do, this will be in line with relevant law and our own policies.

Sage Ireland, trading name of Sage Hibernia Limited. Registered in Ireland at 3096 Lake Drive, City West Business Park, Dublin 24. Registered number 300549.

Kerry,

Thanks for the great response!

I'm really interested in the Duo Test. Do you have some resources or sources that gave you the idea? What kinds of things are the tester observers watching for? If an issue is found, do the testers jump in and get more information or do you wait and do a debrief following the test?

-Doug

>

>
> One practice I like is the "duo test". It's cheap and effective and doesn't require super expertise. Here's how it works. A team of 2 testers conducts the exercise, in which user subjects use the system to perform a specific task and testers observe what happens, looking for usability issues. There are also 2 user subjects, who work together as a team to complete the tasks. This is crucial, because it encourages users to think out loud and discuss problems in ways that thus become more observable. One tester is the scribe and just takes notes about observations. The other tester is the guide who moderates the exercise, answers questions, highlights key observations, and generally keeps things rolling. This is a form of the sort of "heuristic evaluation" that Nielsen describes. You can do a "duo test" with a paper storyboard, a mockup, or an actual system.
>
> > What big mistakes did you learn from early on?
>
> The first time I did a duo test, I tried to video tape it. It was awful and useless. An audio-only recording would have been more useful, but I now think that good notes are best.
>
> Regards,
> Kerry
>

[kerrykimbrough]

> I'm really interested in the Duo Test. Do you have some resources or sources that gave you the idea?

It's an adaptation of various basic techniques for "in the field" UX testing. For another good resource on UX testing, look here: http://www.uie.com/browse/usability_testing/

> What kinds of things are the tester observers watching for?

Good question. You're looking for usability defects -- either definite or potential. But, as always, exactly which UI defects is an important aspect in the design of the test. You may want to focus on specific areas, some of which might be very specific to your app and to your team's UX design rules. It helps to go in with a checklist. You can supplement that with a general usability checklist like the excellent one here: http://www.useit.com/papers/heuristic/heuristic_list.html. This is quite similar to the concept of guided exploratory testing.

> If an issue is found, do the testers jump in and get more information or do you wait and do a debrief following the test?

You recognize an issue from the behavior of the subjects (i.e. confusion, hesitation, mistakes, questions). A good moderator will notice these behaviors and (if necessary) intervene to get the subjects to talk about them. For example: "It looks like this is not making sense. What do you wish the system would do now? Why?". Etc. Notice how important it is to reassure subjects that *they* are not the cause of the problem -- it's the system's fault.

You might spend a bit of time conversing like this, but only enough to be sure you've captured the essence of the problem. But never talk about possible solutions. Leave that for later, in debrief with the dev team. As quickly as possible, return to the main business of watching users work and talk about their work.

[Adrian Howard]

On 7 Oct 2009, at 01:49, kerrykimbrough wrote:
[snip]

>> If an issue is found, do the testers jump in and get more  
>> information or do you wait and do a debrief following the test?
>
> You recognize an issue from the behavior of the subjects (i.e.  
> confusion, hesitation, mistakes, questions). A good moderator will  
> notice these behaviors and (if necessary) intervene to get the  
> subjects to talk about them. For example: "It looks like this is not  
> making sense. What do you wish the system would do now? Why?". Etc.  
> Notice how important it is to reassure subjects that *they* are not  
> the cause of the problem -- it's the system's fault.

[snip]

Another thing to pay attention to - as well as confusion, hesitation,  
mistakes, questions, etc. - is what the user is attempting to do.

For example a user can confidently with no hesitation, etc. do A->B->C-
 >D->E->F to achieve G - completely missing the fact that there was a  
way of just going A->G - but because the option was hidden or non-
obvious they never saw the quicker way of doing it.

Or maybe there was no A->G option - but there could be. Giving you the  
opportunity to save N thousand B->C->D->E->F transitions a day :-)

By focussing on the users goal, rather than the particular way the  
systems lets them achieve the goal at the moment, you can spot  
improvements that somebody habituated to the system doesn't even see  
as a problem.

(which can lead to a whole different set of issues - when making  
something more effective for newbies makes it harder for people used  
to the "old" way of doing it - something that's a hard issue for  
products with an existing user base)

Cheers,

Adrian

--
http://quietstars.com  -  twitter.com/adrianh  -  delicious.com/adrianh

We've started trying to find ways to apply this to a variety of usability questions we've been having with a lot of success. Thank you for a great recommendation.

--- In agile-testing@..., "Coyle, Jim" <jim.coyle@...> wrote:
>
> One thing we do is log usability issues as a category of one of nielsen's 10
> heuristics ... http://www.useit.com/papers/heuristic/heuristic_list.html
> This keeps it objective (as usability can be subjective)
>
>