Usability, Customer Experience & Statistics

10 Things to Know about Usability Problems

Jeff Sauro • July 26, 2011

If you collect nothing else in a usability test it should be a list of problems encountered by users.

It seems so simple yet there is a rich history of how many users you need to test, what constitutes a problem and which method to use.

  1. A usability problem should have a name, description and often a severity rating. Problem severities can be based on a 1-5 or 1-7 scale or assigned according to some criteria like whether the problem.

              1. prevents task completion
              2. causes a significant delay or frustration
              3. has a relatively minor effect on task performance
              4. is a suggestion.

    See Dumas & Redish and Rubin & Chisnell for more.

  2. You should keep track of which user encountered which problem so you can better predict sample sizes, problem discovery rates and the number of undiscovered problems. Use a grid like the following one:

       User 1
     User 2
     User 3
     User 4
     Problem 1
     X  X    
     Problem 2
     Problem 3
    X  X  X  X

  3. If you observe a problem in a usability test with a small sample size, it is more likely that this problem affects a lot of users than just a few.

  4. There are many factors that impact how common usability problems are such as the stage of development and type of interface. In a recent analysis of several datasets I found that business to business application usability problems were about 10 times more common than those found on public facing websites. These are averages from a sample, so the specific context of an application will play a major role in how common usability problems are.

  5. Usability Problems are found in both Heuristic Evaluations and User Testing:  There is some debate on whether User Testing or Heuristic Evaluation reveal more problems or whether HE finds more false positives. When possible, conduct both. Of course, some usability problems are so obvious you don't need to torture any users to know they need to be fixed--and these are the types you hopefully find in Heuristic Evaluations.

  6. Users can report usability problems: Asking users to report what problems they encountered, where they were in the interface and potential solutions can reveal around half the problems as trained usability evaluators. Users aren't substitutes for professional usability evaluators but they can be an easy and cheap option.

  7. Five Users will reveal most of the obvious problems not all problems: Five users will encounter around 85% of problems if the problems tend to affect 31% or more of the users. Five users reveal most of the obvious issues, but not all issues—especially not the ones that tend to affect a small percentage of users (like 10% or fewer). This issue continues to be hotly debated. The controversy is not about the math, it's about the best way to model the percent of problems detected and the percent not found.

  8. Problem severity and frequency are likely independent: It's difficult to separate the number of users that are impacted by a problem from a severity rating, so this will likely generate some correlation, especially when the same evaluators who find the problem rate its severity. This means that more difficult usability problems are just as likely to be encountered as cosmetic ones by users.

  9. Different Evaluators Tend to Find Different Problems: This is due to a number of factors including different definitions of usability problems, different methods, different task scenarios and different users.

  10. Disagreement among usability problem definitions is not unique to usability, many professions, including Radiology and Psychiatry have similar problems. Where judgment is needed and the definition of the "problem" is not specific and measurable, disagreement occurs.

About Jeff Sauro

Jeff Sauro is the founding principal of MeasuringU, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 5 books on statistics and the user-experience.
More about Jeff...

Learn More

You Might Also Be Interested In:

Related Topics

Usability Problems

Posted Comments

There are 6 Comments

January 5, 2017 | weri wrote:

very great post <a href=""></a>rn<a href="">Baju Kebaya Muslim</a> 

November 10, 2016 | felicia wrote:

My life became devastated when my husband sent me packing, after 17 years that we have been together. I was lost and helpless after trying so many ways to make my husband take me back. One day at work, i was absent minded not knowing that my boss was calling me, so he sat and asked me what its was all about i told him and he smiled and said that it was not a problem. I never understand what he meant by it wasn't a problem getting my husband back, he said he used a spell to get his wife back when she left him for another man and now they are together till date and at first i was shocked hearing such thing from my boss. He gave me an email address of the great spell caster who helped him get his wife back, i never believed this would work but i had no choice than to get in contact with the spell caster which i did, and he requested for my information and that of my husband to enable him cast the spell and i sent him the details, but after two days, my mom called me that my husband came pleading that he wants me back, i never believed it because it was just like a dream and i had to rush down to my mothers place and to my greatest surprise, my husband was kneeling before me pleading for forgiveness that he wants me and the kid back home, then i gave mike a call regarding sudden change of my husband and he made it clear to me that my husband will love me till the end of the world, that he will never leave my sight. Now me and my husband is back together again and has started doing pleasant things he hasn't done before, he makes me happy and do what he is suppose to do as a man without nagging. Please if you need help of any kind, kindly contact doctor prince for help and you can reach him via his email at you can also whatsapp him or call him at +2348163241499. 

October 30, 2016 | MIRABEL wrote:

Hi everyone!rnI'm so excited share this testimony here about how i got my ex boyfriend back after a break up. I'm Mirabel 21 yr old from England, my boyfriend of a 4yr just broke up with me and am 30 weeks pregnant.I have cried my self to sleep most of the nights and don’t seem to concentrate during lectures sometimes I stay awake almost all night thinking about him and start to cry all over again.Because of this I end up not having energy for my next day’s classes ,my attendance has dropped and am always in uni and on time.Generally he is a very nice guy ,he ended it because he said we were arguing a lot and not getting along.He is right we’ve been arguing during the pregnancy a lot .After the break up I kept ringing him and telling him I will change.I am in love with this guy and he is the best guy I have ever been with.I’m still hurt and in disbelief when he said he didn’t have any romantic feelings towards me anymore that hurt me faster than a lethal syringe.He texts me now and then mainly to check up on how am doing with the pregnancy,he is supportive with it but it’s not fair on me, him texting me as I just want to grieve the pain and not have any stress due to the pregnancy.i was really upset and i needed help, so i searched for help online and I came across a website that suggested that Dr Purity can help solve marital problems, restore broken relationships and so on. So, I felt I should give him a try. I contacted him and he told me what to do and i did it then he did a spell for me. 28 hours later, my bf came to me and apologized for the wrongs he did and promise never to do it again. Ever since then, everything has returned back to normal. I and my bf are living together happily again.. All thanks to Dr Ahmed . If you need a spell caster that can cast a spell that truly works, I suggest you contact him. He will not disappoint you.This is his E-mail: or Add him up on Whats-app: +2348160153829 

March 22, 2016 | Jack Martin wrote:

Very nice and interesting post. I have even framed few points about Some Common Testing Problems. Hope you would like it -

April 2, 2014 | Mark wrote:

Super useful!rnGreat blog. Avoids fluff and tells me what I need to know! 

March 20, 2013 | Tamirat wrote:

This is very useful 

Post a Comment


Your Name:

Your Email Address:


To prevent comment spam, please answer the following :
What is 3 + 4: (enter the number)

Newsletter Sign Up

Receive bi-weekly updates.
[6634 Subscribers]

Connect With Us

Our Supporters

Use Card Sorting to improve your IA

Loop11 Online Usabilty Testing

Userzoom: Unmoderated Usability Testing, Tools and Analysis


Jeff's Books

Customer Analytics for DummiesCustomer Analytics for Dummies

A guidebook for measuring the customer experience

Buy on Amazon

Quantifying the User Experience 2nd Ed.: Practical Statistics for User ResearchQuantifying the User Experience 2nd Ed.: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to the 2nd Ed. of Quantifying the User ExperienceExcel & R Companion to the 2nd Ed. of Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download