Usability, Customer Experience & Statistics

Compared to What? Making Sense of Customer Experience Metrics

Jeff Sauro • September 28, 2011

These are three of the most important words for anyone trying to make better decisions with data.

I first heard them from Edward Tufte over a decade ago. 

You need a meaningful comparison to turn data into information.

We're often in such a hurry to get the survey out, start the usability test or conduct user interviews that we forget how we'll make sense of all the data. 

Even the best technology or statistical technique cannot substitute for information that puts responses, ratings and revenue in context.

For most measures of customer experience there are at least three good places to start to look for meaningful comparables:
  1. A prior version
  2. An industry average
  3. A leading competitor
Here are some common metrics and some ideas for making them more meaningful:

Task Time: Time is money. But is a task taking too long or is it fast enough? You really need to have comparable data on the same task because subtle differences in task details can make your comparisons less helpful. For example, it takes Southwest Airlines on average 20 minutes to turn a plane around after it lands and have it ready for the next flight. For the same task, it takes United Airlines 35 minutes (see Gladwelll 2010).

Tip: When measuring task time it can be difficult to find prior data for the tasks you're measuring. One trick I use is to have the users attempt the task on the new interface and on the older version during the same evaluation. Alternate which version the participant gets first to minimize sequence effects. At the end of your test you'll know whether all those new features are slowing users down.

Task Difficulty:  It's helpful to use a survey question that's been vetted psychometrically like the Single Ease Question (SEQ). Even with the right question and response options you'll want some comparable. 

Tip: Consider comparing the actual task difficulty rating with the expected task-difficulty rating. You can ask the same users or different users how difficult they think the task will be. The gap in expectations and retrospective accounts can reveal interaction problems.

Overall Ease of Use: What do users think of the overall usability of the application? The System Usability Scale (SUS) is a popular usability questionnaire that's free and short (at 10 items). It provides an overall estimate of usability.

Tip: Take your raw SUS score and convert it to a percentile rank relative to 500 other products or several industry benchmarks. If the product you are testing is consumer/productivity software you can compare your SUS score with 17 other products.

Net Promoter Score: Are customers recommending your software to friends and colleagues? Asking the popular 11-point question is a good start. But is a 25% Net Promoter Score good? 

Tip: Compare your Net Promoter Score to industry benchmarks. If it is consumer software, 25% isn't bad (it's slightly above average) but far below the leader TurboTax.

In your next analysis be sure to ask and answer "Compared to what?"

About Jeff Sauro

Jeff Sauro is the founding principal of MeasuringU, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 5 books on statistics and the user-experience.
More about Jeff...

Learn More

You Might Also Be Interested In:


Posted Comments

Post a Comment


Your Name:

Your Email Address:


To prevent comment spam, please answer the following :
What is 2 + 5: (enter the number)

Newsletter Sign Up

Receive bi-weekly updates.
[6397 Subscribers]

Connect With Us

Our Supporters

Loop11 Online Usabilty Testing

Userzoom: Unmoderated Usability Testing, Tools and Analysis

Use Card Sorting to improve your IA


Jeff's Books

Customer Analytics for DummiesCustomer Analytics for Dummies

A guidebook for measuring the customer experience

Buy on Amazon

Quantifying the User Experience 2nd Ed.: Practical Statistics for User ResearchQuantifying the User Experience 2nd Ed.: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to Quantifying the User ExperienceExcel & R Companion to Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download