Usability, Customer Experience & Statistics

6 Best Practices for Using Numbers to Inform Design

Jeff Sauro • March 22, 2016

Your job title doesn't have to be "researcher" or "statistician" to use data to drive design decisions.

You can apply some best practices even when numbers aren't your best friend.

It's actually easier when you're a designer to enhance your skills with quantitative data than for statisticians to enhance their analytical skills with design principles.

Here are six best practices for using numbers to inform your design efforts that don't require a career change or advanced degree in math.

I'll cover these best practices in London, and more in depth at the Rome UX Boot Camp, and Denver UX Boot Camp.

1. Don't only ask whether people like a design

There's nothing wrong with asking participants their opinion about a design, but you can and should collect much more than what people say they like. You should incorporate what matters to your design into more specific questions. Define what makes a design successful to your organization. Is it:
  • Ease of use
  • Likelihood to use
  • Likelihood to recommend
  • Overall satisfaction
  • Reduced time on task
  • Findability
Decide what metrics matter and measure them.

2. Measure actions and attitudes

The users' experience is comprised of a combination of what people do (actions) and what people think (attitudes). Measure both by having users attempt to do tasks that are most important to them (top-tasks) with a design. Record whether they can do it (effectiveness), and how difficult they found the experience (satisfaction) using the SEQ. You can then compute completion rates.

3. Preference doesn't always equal performance (attitudes don't always equal actions)

When you need to decide between alternative designs, it's good to ask your participants for the design they prefer in order to gain perspective on a design. Instead of making it just a beauty pageant though, have participants attempt to then use each design (in randomized order), rate the experience, and have them pick the one they prefer.

Sometimes the design participants prefer isn't the one they performed best on. When that happens, you need to understand what would cause the disconnect between attitude and action. Often (but not always) the ratings and performance data collected from participants using the design are more indicative of the better design because it's closer to actual use.

4. Have a testable hypothesis

You can make your research efforts a lot more effective when you have a stated hypothesis to test and disprove (or fail to disprove). Your hypotheses don't have to attempt to cure cancer, but they can and should be simple. Here are some hypotheses and corresponding metrics:
  • More people will click the red button than blue (percent of button clicks).
  • Customers will know to scroll down the homepage to view the services (percent of users who scroll down).
  • The new product names will reduce confusion (time to selection, perceived difficulty).
Then use data and scenarios to generate evidence for or against the hypotheses. Having these hypotheses articulated before your research helps ensure you're using the right methods for testing and collecting the right metrics.

5. Use a framework to measure before and after

A good place to start establishing a measurement framework is to know who your users are and what they're trying to do with an interface. But don't stop there. After you define users and tasks, measure how the experience currently is (benchmark), make changes iteratively, and measure how the interface works after your efforts. This helps isolate how your changes make a difference in the experience.


6. Consider sampling error

You might want to test all your users but it's not necessary. When measuring the user experience you'll almost always use a sample of users—and often a small one. With any sample comes sampling error, which is the difference between the sample metrics and population metrics. You want to differentiate real differences from random noise.

You don't need to be a statistician to account for sampling error, but using confidence intervals on your metrics is a simple and effective technique to help you and your stakeholders understand how much uncertainty there is in your estimates and compare your initial designs to your later designs.


About Jeff Sauro

Jeff Sauro is the founding principal of MeasuringU, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 5 books on statistics and the user-experience.
More about Jeff...


Learn More



You Might Also Be Interested In:

Related Topics

Methods
.

Posted Comments

Newsletter Sign Up

Receive bi-weekly updates.
[6505 Subscribers]

Connect With Us

Our Supporters

Userzoom: Unmoderated Usability Testing, Tools and Analysis

Usertesting.com

Use Card Sorting to improve your IA

Loop11 Online Usabilty Testing

.

Jeff's Books

Customer Analytics for DummiesCustomer Analytics for Dummies

A guidebook for measuring the customer experience

Buy on Amazon

Quantifying the User Experience 2nd Ed.: Practical Statistics for User ResearchQuantifying the User Experience 2nd Ed.: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to the 2nd Ed. of Quantifying the User ExperienceExcel & R Companion to the 2nd Ed. of Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download

.
.
.