Usability, Customer Experience & Statistics

The Importance of Evaluating UX

Jeff Sauro • November 3, 2015

Maybe you've seen the ketchup bottles.

It's been a popular meme on LinkedIn. The bottles are intended to be a clever way to explain the difference between UI (user interface) and UX (user experience).

The classic bottle of ketchup is the user interface. The ketchup bottle introduced in 2002 with its characteristic inverted bottle and lid is supposed to be the user experience.

While people spreading the meme have good intentions, this example illustrates why there's still confusion over the difference between UI and UX. Both bottles provide a user interface and both have a user experience.

The intended message here is that the newer ketchup bottle has a better user experience because it's focused on a common user problem of ketchup being stuck in the bottle.
 
And maybe it does have a better user experience. But a user experience is not better just because it's been redesigned--even if it's been redesigned for the user in mind.

For example, if you use the upside-down ketchup bottles for a while, goopy ketchup begins to build up in the lid. Even worse, the upside-down orientation puts pressure on the lid so when you open it, ketchup sprays on your clothes!


 
A user experience is better because improvements to the interface are measurably better. Maybe it allows users to more effectively complete a task in a way that's quicker or more satisfying, or accomplish something that wasn't possible before.

Now, I suspect the good folks at Heinz went through several rounds of empirical research to develop the bottle and maybe found it a measurably better experience, despite the messy problem.  Showing the research team and the report is a lot less picturesque than the bottle. But it's the process that gets to the bottle that makes the user experience better.

So while we all want to have a better ketchup experience, few of us has the responsibility to actually design and evaluate ketchup bottles. However, you can apply a simple framework for evaluating the user experience for whatever you design, be it software, websites, mobile phones, and yes, even ketchup bottles.

Framework to Evaluate and Improve the User Experience

The framework to evaluate and improve the experience for your users is relatively simple and flexible; we use it extensively at MeasuringU. It's based on the Six Sigma DMAIC methodology that's been successfully applied across thousands of organizations and disparate products.


  1. Define your outcome measures.

    Your metrics should match your research goals. Findability, completion rates, likelihood to recommend, and purchase rate are all good items to measure. Or maybe ketchup pour accuracy, pour time, and spill factors?


  2. Define the users and what they're trying to accomplish.

    Who are the users and what are they trying to do? Use a combination of true-intent surveys to understand users and a top-task analysis to separate the many trivial tasks from the vital few. When it comes to ketchup, it's safe to say it's about the pour (or squirt). And you'll want to avoid those few ketchup haters out there in your sample.

  3. Measure the baseline.

    You don't know if you've improved the experience unless you know where it is before changes are made. Measure first using the existing product experience with a sample of users attempting the top tasks and collect the metrics you identified.


  4. Make your changes.

    Implement your new design, feature, branding, or bug fix. Think iteratively here. It's better to fail fast. I wonder how many iterations Heinz went through before deciding on the specific bottle. Would a skinny top ketchup bottle work better, like the one pictured below?  What new problems does it create?



  5. Measure after your redesign.

    Don't forget to measure again. It's like the bread on the evaluation sandwich—you need two slices around the meat! It may seem like your changes are guaranteed to make things better, but did they and by how much? Did you introduce new problems in the process? Here's where you see whether the new ketchup bottle design's potential shortcomings are worth the putative improvement in getting the ketchup out of the bottle. If they are, the measures should show it.

Summary

Every product, website, or design has a user interface. If it's used, it has a user experience. What differentiates a good user experience from a bad one is not based on how many awards and accolades it gets (that's nice of course); instead, it's based on superior measurable outcomes.

Using the framework of defining metrics, users, and tasks, and measuring before and after changes helps ensure the user experience is quantifiably better.  Now, please someone pass me the ketchup!
 

About Jeff Sauro

Jeff Sauro is the founding principal of MeasuringU, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 5 books on statistics and the user-experience.
More about Jeff...


Learn More

UX Bootcamps: Rome: June 20-22, 2016 and Denver: Aug 17-21, 2016
Quantifying the User Experience Half Day Seminar: London: June 15th, 2016 and Chicago: July 15th, 2016


You Might Also Be Interested In:

Related Topics

Methods
.

Posted Comments

Newsletter Sign Up

Receive bi-weekly updates.
[5943 Subscribers]

Connect With Us

UX Bootcamp

Rome:June 20-22 & Denver: Aug. 17-19 2016

3 Days of Hands-On Training on UX Methods, Metrics and Analysis.Learn More

Our Supporters

Usertesting.com

Use Card Sorting to improve your IA

Loop11 Online Usabilty Testing

Userzoom: Unmoderated Usability Testing, Tools and Analysis

.

Jeff's Books

Customer Analytics for DummiesCustomer Analytics for Dummies

A guidebook for measuring the customer experience

Buy on Amazon

Quantifying the User Experience: Practical Statistics for User ResearchQuantifying the User Experience: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to Quantifying the User ExperienceExcel & R Companion to Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download

.
.
.