Usability, Customer Experience & Statistics

Sample Size Calculator for Discovering Problems in a User Interface

Jeff Sauro • October 1, 2006

Sample Size from an estimate of Problem Occurrence (p)

If the probability of detecting a UI problem is known in advance, use this portion of the calculator to estimate the total number of users needed to uncover on average the specified percentage of problems (e.g. 90%). The calculator is based on the binomial probability formula.
Discover of all Problems.
Problem Occurrence (values between 0 and 1)


Estimate Problem Occurrence (p) then Sample Size
This portion of the calculator first builds an estimate of the probability of detecting a UI problem (from sample data). It then produces an estimate of the number of users needed to discover the specified percent of total problems. It uses the Good-Turing and Normalization procedure as outlined by Lewis (2001) and further discussed in (Turner, Lewis & Nielsen 2006).

Discover of all Problems.
Total participants
Problems Discovered:

Lewis, James (2001) "Evaluation of Procedures for Adjusting Problm-Discovery Rates Estimated from Small Samples" in The International Journal of Human-Computer Interaction 13(4) p. 445-479

Turner, C. W., Lewis, J. R., and Nielsen, J. (2006). Determining usability test sample size. In W. Karwowski (Ed.), International Encyclopedia of Ergonomics and Human Factors (pp. 3084-3088). Boca Raton, FL: CRC Press.

About Jeff Sauro

Jeff Sauro is the founding principal of MeasuringU, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 5 books on statistics and the user-experience.
More about Jeff...

Learn More


Posted Comments

There are 9 Comments

September 25, 2014 | roxana wrote:

Unfortunately, these calculators no longer work. Could you maybe update the code? Thanks! 

September 5, 2012 | Razeen Davids wrote:

Hi Jeff, please check the "Estimate Problem Occurrence (p) then Sample Size" calculator, not working.


February 22, 2012 | Jeff Sauro wrote:

Great question. If we assume 30% of users actually would have a problem with part of the interface, then you'll have a good chance of seeing that AT LEAST once in your usability test of 5 users.

A problem is generally considered a problem if it happens at least one time and that's how the calculator works. Determining if a problem is more than minor is an issue of judgment and not math.

For example, in a recent benchmarking study of a rental car website, we watched as one user was ready to rent the car, but then switched the payment option on the checkout screen to "Pay Now" instead of "Pay at the Counter." In making this change, the price didn't change (as it did on the previous page).

This only happened to 1 out of 5 users. Perhaps this is a bug or it's working as intended. It could take a lot to fix this issue and maybe the development team will only fix it if they know enough users will actually do this (so it's not just a fluke). Given just 5 total tested, we can be 95% confident between 2% and 64% of users will have the problem. Definite uncertainty, but highly unlikely that say only 1% of users would have the problem. If the problem impacts 5 out of 5, it's likely at least 60% of the user population would also have the problem.

This touches on the issues of problem frequency and severity. A problem can happen to 5/5 and it seems more severe simply because of its frequency, but its impact can be minor (like a small typo) or a problem can impact 1/5 users and its impact can be major (loss of a sale).

As a matter of strategy you can certainly have a threshold where something is only considered a problem if it happens to AT LEAST 2 users, instead of AT LEAST 1 user. This increases the sample size needed and you can see Table 1 here:


February 22, 2012 | Lucas wrote:

Hey Jeff,
I am a big fan of your blog and after reading this post I kept wondering... in the first example "You would need to test on average 5.3189045575676 users to discover 85% of UI problems given the occurrence of a problem is 30%." How many users out of those 5 (.3189...) would have experience the same problem so that you may consider it a "30%'s problem"?
I mean, if the same problem occurs to 5 out of 5 of your users I guess that's not the same as if that problem occured to only 1 out of 5, am I right?
Does that problem have to occur to ALL the 5 users or if it happened to just 1 that´s considered enough?
Thank you in advance! 

February 2, 2012 | Ivan wrote:

the "Estimate Problem Occurrence (p) then Sample Size" calculator doesn't work now.
Uncaught TypeError: Object [object Object] has no method 'prettyGallery' 

June 17, 2009 | Davant G Bryant wrote:

6. Why is population shape of concern when estimating a mean? What does sample size have to do
with it? 

April 30, 2008 | Raul R Wells wrote:

Very useful, I could not move the bar, so it reported 50% against a 100% I wished to mark 

January 4, 2008 | John Romadka wrote:

It might help if you could give a real world example of how these calculators could be used. Because I think I get it, but an example would make it more concrete. 

January 4, 2008 | John Romadka wrote:

I wonder if you could add a little contextual help for the Problem Occurance field (".30"). Specifically, when would I use a number closer to 0 or a number closer to 1. 

Post a Comment


Your Name:

Your Email Address:


To prevent comment spam, please answer the following :
What is 2 + 4: (enter the number)

Newsletter Sign Up

Receive bi-weekly updates.
[6216 Subscribers]

Connect With Us

Our Supporters

Loop11 Online Usabilty Testing

Use Card Sorting to improve your IA

Userzoom: Unmoderated Usability Testing, Tools and Analysis


Jeff's Books

Customer Analytics for DummiesCustomer Analytics for Dummies

A guidebook for measuring the customer experience

Buy on Amazon

Quantifying the User Experience: Practical Statistics for User ResearchQuantifying the User Experience: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to Quantifying the User ExperienceExcel & R Companion to Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download