Usability, Customer Experience & Statistics

Asking the Right User Experience Questions

Jeff Sauro • December 5, 2012

Few things are more revealing than watching just a few users attempt tasks on a website or software.

But while you're watching users you should ask them some key questions to help put the observations into perspective.

Here are eight recommendations for helping quantify both attitudes and put the insightful observations into context.

Software Usability-- Use the System Usability Scale
(SUS): This venerable 10 item instrument is the most used questionnaire for measuring perceptions of usability. It's short, reliable and free (don't we wish everything was like this?). It's technology agnostic, which means you can use it on software, mobile apps, hardware, Voice Response Systems and websites.

Websites Quality-- Use the SUPR-Q. The 13 item Standardized Universal Percentile Rank Questionnaire (SUPR-Q) provides four essential measures of website quality: usability, loyalty, credibility and appearance. In fact, just four items can accurately predict the System Usability Scale. Raw scores can be associated with a database of 200 other websites.

Task Difficulty-- Use the SEQ:  After users attempt a task in a usability test, ask them how difficulty or easy they thought it was on this single 7 point scale. We've found that this single question performed about as well as more complicated and lengthy questionnaires. User time is limited so the shorter the better. In this case the SEQ is both short and good.

Task Confidence:  Users might find ways of getting through task scenarios but just aren't quite sure they did things correctly. In such cases, getting a measure of confidence can help tell if the task completion rate is masking some underlying issues. We also use confidence along with completion rates to generate Disaster Rates: when users are extremely confident and yet fail the task.

Customer Loyalty--Use the Net Promoter Score:  Word of mouth is the engine of viral growth and the 11 point Net Promoter Score has become the defacto measure of customer and user loyalty. If users are saying positive things about your software, website or other product to a friend then growth prospects are good. Conversely, if others have had bad experiences and are telling others then bad news is spreading fast. In both cases you'll want to have a good measure of customer loyalty to act upon.

Product Usefulness-- Use the Technology Acceptance Model:  You can have the most usable and elegant product but if no one feels a need to use it, not much else matters. The questionnaire based on the TAM is 20 items, 10 provide a measure of usefulness which can be used as a measure over time or when comparing products with different features. The other 10 items are similar to the System Usability Scale, so when measuring usability use the SUS.

Why?: The items above use closed-ended rating scales. This is helpful for putting problems and actions into context but the numbers alone don't provide much insight into what the problem is. When possible, ask users to provide in their own words, their thoughts about the product. We find this especially helpful when we get lower ratings on task-difficulty and Net Promoter Scores

What one thing should be fixed?:  Once you're asking what the problem is, use the opportunity to ask users what they would fix in the software or website. While we can't expect users to be designers or do the job for us, it's good to capture those problems that are at the top-of-mind of users. You can't fix everything, but usually there are 1 or 2 things that are not too difficult to fix and the improvement would benefit a lot of users.

About Jeff Sauro

Jeff Sauro is the founding principal of MeasuringU, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 5 books on statistics and the user-experience.
More about Jeff...

Learn More

You Might Also Be Interested In:

Related Topics

Usability, UX

Posted Comments

There are 1 Comments

October 8, 2013 | Pav wrote:

Is this a best user experience for conditional form elements to to assign a mutually exclusive selection with user selecting any number of options.rnRadio buttons(mutually exclusive) having checkboxes (selecting any number of items) 

Post a Comment


Your Name:

Your Email Address:


To prevent comment spam, please answer the following :
What is 2 + 1: (enter the number)

Newsletter Sign Up

Receive bi-weekly updates.
[6389 Subscribers]

Connect With Us

Our Supporters

Loop11 Online Usabilty Testing

Userzoom: Unmoderated Usability Testing, Tools and Analysis

Use Card Sorting to improve your IA


Jeff's Books

Customer Analytics for DummiesCustomer Analytics for Dummies

A guidebook for measuring the customer experience

Buy on Amazon

Quantifying the User Experience 2nd Ed.: Practical Statistics for User ResearchQuantifying the User Experience 2nd Ed.: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to Quantifying the User ExperienceExcel & R Companion to Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download