Usability, Customer Experience & Statistics

25 Rules & Recommendations for the UX Researcher

Jeff Sauro • February 24, 2015

User research is not a hard science.

But like all behavioral sciences, there are principles, best-practices, rules and recommendations.

Across dozens of tutorials, five books, many articles and blogs, boot camps, and discussions with both seasoned and new UX professionals, I've noticed a number of common problems and themes related to measuring the customer experience.

Here's a list of 25 rules and recommendations that people tend to find most helpful.

  1. Mix methods, metrics, and tools. Don't always use the same ones. And use them in combination: think and instead of or.

  2. It's much easier to prove that something is not usable than to prove that it is usable. Small samples can quickly show that an experience is terrible. However, just because a few participants don't have a problem doesn't mean the experience is great.

  3. Preference does not equal performance, but the two often go hand in hand. Measure what users do, ask them what they prefer, and then reconcile any difference.

  4. No single sample size (not even five) works for every study.

  5. If a problem affects 100% of your user population, testing with one user will expose that problem.

  6. Randomize. Wherever you can, alternate the order of tasks, products, questions, and response options. Randomization can often minimize even major flaws in research designs.

  7. To cut your margin of error in half, quadruple your sample size.

  8. A 5-point rating scale is usually better than a 3-point scale, a 7-point scale is slightly better than a 5-point scale, and an 11-point scale is ever-so-slightly better than a 7-point scale. Increasing a scale to over 11 points adds little value, especially if you're using more than one item in a questionnaire, as with the SUS or SUPR-Q.

  9. The number of scale points doesn't matter much. If you find yourself arguing about the number of points in a response scale, you've probably already spent too much time on the topic. Ditto for arguments about having neutral responses and whether the positive responses should be on the left or right. Pick a scale and stick with it over time.

  10. No one in a company owns the user data. Everyone owns the user data. Everyone can contribute, everyone is responsible and everyone should care.

  11. While Mad Men–style one-way mirrors in labs may seem cool and may even impress customers, digital technology has largely replaced the need for noisy, dark, cramped observation rooms next to the usability lab.

  12. Know who your users are. While personas serve a purpose, you don't need fancy, life-size cardboard cut-outs. You do need to determine which behaviors, goals, and attitudes define the people who use your website or software and who buys your products—and choose your test participants accordingly.

  13. Know what users are trying to do, and you'll avoid one of the biggest disconnects we see. Often, product stakeholders are sure that users are doing one thing while the data reveals that they're doing something else. Conduct a top-task analysis; it's one of the best bangs for your research buck.

  14. Many design differences are not noticeable to users. When very large sample sizes detect no differences in attitudes and behaviors, it means that the design changes don't greatly impact users' goals.

  15. What users say and what they do aren't always the same, but they often are. Measure both.

  16. Usability doesn't exist. It's a reified construct. There's no usability yardstick. We rely instead on the outcomes of poor usability: longer times, more failed tasks, and low ratings. We should however aim for the yardstick by continually improving our measurement methods.

  17. Usability isn't the only thing users care about on software and websites. Utility, appearance, credibility, and accessibility are among the main factors you must consider.

  18. Measures of usability are not invariant. They change based on the tasks, users, evaluators, and other variables.

  19. Be open-minded. Don't dismiss new tools, techniques, or metrics without trying them out.

  20. Think lean. By the time a report has been revised and presented, you have to know what the problems are and, ideally, how to solve them. While the 1980s were a great decade for music and big hair, be sure your reports aren't stuck there.

  21. Jakob Nielsen and Jared Spool can both be right. Don't get caught up in picking one UX celebrity over another. While there are different perspectives on how to conduct and improve user research, there's much more agreement than disagreement.

  22. There's no killer app to solve all your usability problems. Don't be wedded to any tool.

  23. What you see is not always what you get. Eye-trackers are cool and powerful tools. Executives love the heat maps. But just because participants look at a design element or copy, it doesn't mean they perceive, understand, or remember it. Many research questions can be answered without them.

  24. What you get is what you see. Different evaluators tend to see different usability problems—even if they watch the same videos of the same users!

  25. There are many paths up the same mountain. Usability is less about generating consistent results between evaluators than it is about finding and fixing problems.

About Jeff Sauro

Jeff Sauro is the founding principal of MeasuringU, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 5 books on statistics and the user-experience.
More about Jeff...

Learn More

UX Bootcamps: Rome: June 20-22, 2016 and Denver: Aug 17-21, 2016
Quantifying the User Experience Half Day Seminar: London: June 15th, 2016 and Chicago: July 15th, 2016

You Might Also Be Interested In:

Related Topics


Posted Comments

Newsletter Sign Up

Receive bi-weekly updates.
[5960 Subscribers]

Connect With Us

UX Bootcamp

Rome:June 20-22 & Denver: Aug. 17-19 2016

3 Days of Hands-On Training on UX Methods, Metrics and Analysis.Learn More

Our Supporters

Userzoom: Unmoderated Usability Testing, Tools and Analysis

Loop11 Online Usabilty Testing

Use Card Sorting to improve your IA


Jeff's Books

Customer Analytics for DummiesCustomer Analytics for Dummies

A guidebook for measuring the customer experience

Buy on Amazon

Quantifying the User Experience: Practical Statistics for User ResearchQuantifying the User Experience: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to Quantifying the User ExperienceExcel & R Companion to Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download