Jeff Sauro • January 24, 2012

There is an erroneous perception in the UX community that if your method is qualitative, then numbers somehow cannot or should not be used.These perceptions come from an informal practice that stems back to the beginning of the usability profession and continues through training programs and some UX experts.

Unfortunately, this perception is misguided and can prevent perfectly good data from being used to gain accurate views of the user experience.

Qualitative data can in fact be converted into quantitative measures even if it doesn't come from an experiment or from a large sample size.

The distinction between a qualitative study and quantitative study is a false dichotomy. It doesn't cost more money to quantify or use statistics. It just takes some training and confidence--like any method or skill.

Here are five examples of how you can take common qualitative approaches to assessing the user experience and convert them into numbers which can then be treated with a range of statistical procedures.

- Converting a usability problem into a frequency: The quintessential usability activity is watching users attempt realistic tasks and identifying what in the interface is causing problems.

Simply categorize the problems, count the frequency, then use confidence intervals to estimate how common the problems likely are in the entire user population. For example, if 3 out 11 users had a problem downloading the correct software product from a website, then we can be 95% confident at least 9% of all users would also have the problem (use the free web calculator or download the problem frequency calculator). It doesn't cost more money to generate those confidence intervals. This process also allows you to generate more accurate sample size estimates. - What problems are customers having?: It is sometimes difficult for customers to identify what they need in a product and where the shortfalls are. One effective approach is ethnographic research (a qualitative method), observing customers in their own setting encountering and solving problems.

Observe the problems customers encounter, categorize and count them. Then estimate the percent of all customers that likely share this behavior or problem to help prioritize product features. You can then estimate how many customers you need to visit based on the frequency of these issues. - Why is the product not being recommended? When using the Net Promoter Score, it's valuable to ask open-ended, follow-up questions, especially for Detractors, such as, "Briefly describe why you gave the rating." Take the list of open-ended comments and group them into categories (content analysis). Count the occurrences, create a percentage of all comments, graph them and throw in some confidence intervals for good measure.
- Why was that task so difficult?: I recommend asking just a single question after users attempt a task in an informal Steve Krug usability test. If a user provides a low rating (below a 5), ask them to briefly explain why they gave a low rating. Take these open-ended comments, categorize them and add up the frequency in each group. This process can help you and your stakeholders make more informed decisions about the likely causes of the trouble. Figure 1 below shows an example of the comments from a recent usability test.

Figure 1: Categorized comments about why the task was difficult. Error bars are 90% confidence intervals, N= 106. - Combining Net Promoter Scores and comments: A powerful way of making qualitative, open-ended comments more actionable is to combine them with a closed-ended question, like the Net Promoter Score. For example, quantify what users say they would improve on a website, then show what these customer's Net Promoter Scores are.

An example is shown in Figure 2 below. There were 110 comments in total, but to quickly identify what to focus on, we can see that comments related to website navigation and product filters are both high in frequency and come from users that are likely generating negative word of mouth (notice the negative NPS). In contrast, design/layout comments and advertisements while high in frequency appear to be minor issues for the users.

Figure 2: Combining open-ended comments about what to fix on a website with those users' Net Promoter Scores. Comments related to Navigation and Product filters are both high in frequency and come from the more dissatisfied users. Error bars are 90% confidence intervals.

But once you've picked the low hanging fruit of a neglected interface, the benefits of structuring your activities and results lend themselves to quantification, where you can derive more meaning from your methods.

The advantage of converting qualitative data into quantitative data is that the source of qualitative data--a direct encounter of the user's experience--can reveal nuances in usability, perhaps otherwise missed in more formal quantitative experiments and surveys.

Not only can qualitative data be categorized into quantities, but it can prompt further questions and discovery for usability improvement.

6 Ways to Visualize Statistical Significance

Reflecting on the One-Way Mirror

A Brief History of the Magic Number 5 in Usability Testing

Confidence Interval Calculator for a Completion Rate

97 Things to Know about Usability

Nine misconceptions about statistics and usability

5 Examples of Quantifying Qualitative Data

Should you use 5 or 7 point scales?

8 Ways to Show Design Changes Improved the User Experience

Does better usability increase customer loyalty?

How common are usability problems?

Why you only need to test with five users (explained)

How to Conduct a Usability test on a Mobile Device

10 Things to Know about Usability Problems

.

Customer Analytics for DummiesA guidebook for measuring the customer experience Buy on Amazon | |

Quantifying the User Experience 2nd Ed.: Practical Statistics for User ResearchThe most comprehensive statistical resource for UX Professionals Buy on Amazon | |

Excel & R Companion to the 2nd Ed. of Quantifying the User ExperienceDetailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R Buy on Amazon | Download | |

A Practical Guide to the System Usability ScaleBackground, Benchmarks & Best Practices for the most popular usability questionnaire Buy on Amazon | Download | |

A Practical Guide to Measuring Usability72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software Buy on Amazon | Download |

.

.

.