Beta Testing For Usability?

Hello, everyone!

We're about to roll out a few applications to a beta test group and I was wondering what techniques you've all used to get useful feedback about usability during beta test phases.

My current plan is:

1. Interview testers before they get the beta release. The purposes of this interview are to level set about what the testers expect the application will do for them, what they are currently using to fill the gap (internet sites, etc), and what they hope the beta application does not do.

2. Weekly polling to see how well or not the beta is living up to expectations for each tester.

3. Weekly ethnographical studies with select testers (preferrably those who seem frustrated with the application).

4. Bi-weekly publishing of test results with recommendations for how to proceed on issues found.

Have you used other approaches or techniques during betas to get great "real world" feedback on usability? What techniques had the highest yield?

[Kaleb Pederson-2]

On Mon, Nov 2, 2009 at 12:58 PM, doug.gorman@...
<doug.gorman@...> wrote:
> Hello, everyone!
>
> We're about to roll out a few applications to a beta test group and I was wondering what techniques you've all used to get useful feedback about usability during beta test phases.

Nice question.

> Have you used other approaches or techniques during betas to get great "real world" feedback on usability? What techniques had the highest yield?

Depending on the type of application, a true real-world indicator of
the effectiveness of various UI changes would be to A/B test them
(http://en.wikipedia.org/wiki/A/B_testing).  Assuming you're working
with a web application, you should be able to send a certain
percentage of users (possibly pre-screened and filtered) through the
beta page(s) to determine the overall affect on the target metrics. If
a single change positively effects the target metrics, like conversion
or lead generation, then it's probably a good change and likely makes
the application more usable.

--
Kaleb Pederson

Blog - http://kalebpederson.com
Twitter - http://twitter.com/kalebpederson

[Miinalainen, Petteri]

Hi,

I've seen that a free form forum can also help to find solutions.

For example, someone is having trouble with new feature and asks for help. That often leads to a discussion with multiple solutions to problem and brings several opinions and related issues to table at once.
Other than that, your list seems to be quite comprehensive.

Petteri

--- In agile-usability@..., "doug.gorman@..." <doug.gorman@...> wrote:

>
> Hello, everyone!
>
> We're about to roll out a few applications to a beta test group and I was wondering what techniques you've all used to get useful feedback about usability during beta test phases.
>
> My current plan is:
>
> 1. Interview testers before they get the beta release. The purposes of this interview are to level set about what the testers expect the application will do for them, what they are currently using to fill the gap (internet sites, etc), and what they hope the beta application does not do.
>
> 2. Weekly polling to see how well or not the beta is living up to expectations for each tester.
>
> 3. Weekly ethnographical studies with select testers (preferrably those who seem frustrated with the application).
>
> 4. Bi-weekly publishing of test results with recommendations for how to proceed on issues found.
>
> Have you used other approaches or techniques during betas to get great "real world" feedback on usability? What techniques had the highest yield?
>

Thanks, Kaleb. I've recently found some great articles on A/B testing and it's use in product and usability development.

This was an excellent one: http://blog.mixpanel.com/ab-testing-to-increase-user-engagement

I definitely think we'll try this approach.

--- In agile-usability@..., Kaleb Pederson <kaleb.pederson@...> wrote:

>
> On Mon, Nov 2, 2009 at 12:58 PM, doug.gorman@...
> <doug.gorman@...> wrote:
> > Hello, everyone!
> >
> > We're about to roll out a few applications to a beta test group and I was wondering what techniques you've all used to get useful feedback about usability during beta test phases.
>
> Nice question.
>
> > Have you used other approaches or techniques during betas to get great "real world" feedback on usability? What techniques had the highest yield?
>
> Depending on the type of application, a true real-world indicator of
> the effectiveness of various UI changes would be to A/B test them
> (http://en.wikipedia.org/wiki/A/B_testing).  Assuming you're working
> with a web application, you should be able to send a certain
> percentage of users (possibly pre-screened and filtered) through the
> beta page(s) to determine the overall affect on the target metrics. If
> a single change positively effects the target metrics, like conversion
> or lead generation, then it's probably a good change and likely makes
> the application more usable.
>
> --
> Kaleb Pederson
>
> Blog - http://kalebpederson.com
> Twitter - http://twitter.com/kalebpederson
>