Usability, Customer Experience & Statistics

10 Things to Know about Usability Problems

Jeff Sauro • July 26, 2011

If you collect nothing else in a usability test it should be a list of problems encountered by users.

It seems so simple yet there is a rich history of how many users you need to test, what constitutes a problem and which method to use.

  1. A usability problem should have a name, description and often a severity rating. Problem severities can be based on a 1-5 or 1-7 scale or assigned according to some criteria like whether the problem.

              1. prevents task completion
              2. causes a significant delay or frustration
              3. has a relatively minor effect on task performance
              4. is a suggestion.

    See Dumas & Redish and Rubin & Chisnell for more.

  2. You should keep track of which user encountered which problem so you can better predict sample sizes, problem discovery rates and the number of undiscovered problems. Use a grid like the following one:

       User 1
     User 2
     User 3
     User 4
     Problem 1
     X  X    
     Problem 2
     Problem 3
    X  X  X  X

  3. If you observe a problem in a usability test with a small sample size, it is more likely that this problem affects a lot of users than just a few.

  4. There are many factors that impact how common usability problems are such as the stage of development and type of interface. In a recent analysis of several datasets I found that business to business application usability problems were about 10 times more common than those found on public facing websites. These are averages from a sample, so the specific context of an application will play a major role in how common usability problems are.

  5. Usability Problems are found in both Heuristic Evaluations and User Testing:  There is some debate on whether User Testing or Heuristic Evaluation reveal more problems or whether HE finds more false positives. When possible, conduct both. Of course, some usability problems are so obvious you don't need to torture any users to know they need to be fixed--and these are the types you hopefully find in Heuristic Evaluations.

  6. Users can report usability problems: Asking users to report what problems they encountered, where they were in the interface and potential solutions can reveal around half the problems as trained usability evaluators. Users aren't substitutes for professional usability evaluators but they can be an easy and cheap option.

  7. Five Users will reveal most of the obvious problems not all problems: Five users will encounter around 85% of problems if the problems tend to affect 31% or more of the users. Five users reveal most of the obvious issues, but not all issues—especially not the ones that tend to affect a small percentage of users (like 10% or fewer). This issue continues to be hotly debated. The controversy is not about the math, it's about the best way to model the percent of problems detected and the percent not found.

  8. Problem severity and frequency are likely independent: It's difficult to separate the number of users that are impacted by a problem from a severity rating, so this will likely generate some correlation, especially when the same evaluators who find the problem rate its severity. This means that more difficult usability problems are just as likely to be encountered as cosmetic ones by users.

  9. Different Evaluators Tend to Find Different Problems: This is due to a number of factors including different definitions of usability problems, different methods, different task scenarios and different users.

  10. Disagreement among usability problem definitions is not unique to usability, many professions, including Radiology and Psychiatry have similar problems. Where judgment is needed and the definition of the "problem" is not specific and measurable, disagreement occurs.

About Jeff Sauro

Jeff Sauro is the founding principal of MeasuringU, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 5 books on statistics and the user-experience.
More about Jeff...

Learn More

You Might Also Be Interested In:

Related Topics

Usability Problems

Posted Comments

There are 3 Comments

March 22, 2016 | Jack Martin wrote:

Very nice and interesting post. I have even framed few points about Some Common Testing Problems. Hope you would like it -

April 2, 2014 | Mark wrote:

Super useful!rnGreat blog. Avoids fluff and tells me what I need to know! 

March 20, 2013 | Tamirat wrote:

This is very useful 

Post a Comment


Your Name:

Your Email Address:


To prevent comment spam, please answer the following :
What is 1 + 5: (enter the number)

Newsletter Sign Up

Receive bi-weekly updates.
[6389 Subscribers]

Connect With Us

Our Supporters

Use Card Sorting to improve your IA

Loop11 Online Usabilty Testing

Userzoom: Unmoderated Usability Testing, Tools and Analysis


Jeff's Books

Customer Analytics for DummiesCustomer Analytics for Dummies

A guidebook for measuring the customer experience

Buy on Amazon

Quantifying the User Experience 2nd Ed.: Practical Statistics for User ResearchQuantifying the User Experience 2nd Ed.: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to Quantifying the User ExperienceExcel & R Companion to Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download