Jeff Sauro • April 17, 2012

One of the simplest ways to measure any event is a binary metric coded as a 1 or 0.Such a metric represents the presence or absence of just about anything of interest: Yes/ No, Pass/ Fail, Purchase/No Purchase, On/Off.

Fundamentally the binary system is at the heart of computing as we know it. It also plays a critical role in user research.

Binary measures can even apply to traditionally qualitative research such as noting whether a user mentions distrust of sales people or marketing materials in an interview.

By adding up all the 1's and dividing by the total number observed you get a proportion. So if 5 out of 10 users mention distrusting marketing materials in an interview you get a proportion of 5/10 = .5 which is often expressed as a percentage 50%.

You can use proportions to help make data driven decisions just about anywhere: Which design converts more? Which product is preferred? Does the new interface have a higher completion rate? What proportion of users had a problem registering?

When it comes to comparing independent proportions you'll need to conduct a statistical test called the 2 proportion test (which is equivalent to the Chi-Square test).

We talk a lot about comparing two proportions in our book Quantifying the User Experience (Chapter 5) and recommend a slight adjustment to the typical formulas you might be familiar with so it works for small and large sample sizes.

To make it easier to compare proportions, I've created a simple online web-calculator which will do the statistical calculations for you. Just enter two proportions to see if the difference between them is more likely due to chance or more likely a legitimate difference.

Here are 6 examples of proportions and the statistical results to get you thinking:

- Completion Rates: If 11 out 12 users complete a task on Design A and only 5 out of 10 can complete the same task on Design B, then we can be 97% confident more users can complete the task on Design A.
- Conversion Rates: A large blue button was shown to 455 users and 37 (8%) purchased a product. A large red button was shown to 438 different users and 22 purchased the product (5%). There is a 94% chance the blue button will sell more products.
- Problem Occurrence: 4 out of 7 users received at least one error message when entering alerts and notifications into their profile on a credit card website. After a redesign, 1 out of 7 had at least one error. There is an 89% chance the number of errors has been reduced when setting account alerts.
- Proportion Recommending: 89 out of 100 (93%) customers said they recommended Smart Phone A to a friend in the last year compared to 67 out of 93 (72%) for Smart Phone B. There is a 99.7% chance this retroactive recommend rate is different.
- Proportion detracting: Prior to the change in the return policy, 49 out of 100 (49%) customers surveyed were detractors. After the change in policy 40 out of 96 (42%) were. There is about a 69% chance the difference is not due to chance (good, but not overwhelming evidence).
- Proportion that completed a task in less than 30 seconds: 4 out of 9 users could add a new contact in CRM application A in less than 30 seconds. Eleven out of 12 could on CRM application B. There is a 97% chance if we tested all users, more would complete the tasks on App B.

Getting Started Finding the Right Sample Size

The Essentials of a Contextual Inquiry

5 Examples of Quantifying Qualitative Data

How common are usability problems?

Why you only need to test with five users (explained)

Nine misconceptions about statistics and usability

97 Things to Know about Usability

The Five Most Influential Papers in Usability

What five users can tell you that 5000 cannot

10 Things to Know about Usability Problems

Should you use 5 or 7 point scales?

8 Ways to Show Design Changes Improved the User Experience

Does better usability increase customer loyalty?

How to Conduct a Usability test on a Mobile Device

.

Customer Analytics for DummiesA guidebook for measuring the customer experience Buy on Amazon | |

Quantifying the User Experience: Practical Statistics for User ResearchThe most comprehensive statistical resource for UX Professionals Buy on Amazon | |

Excel & R Companion to Quantifying the User ExperienceDetailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R Buy on Amazon | Download | |

A Practical Guide to the System Usability ScaleBackground, Benchmarks & Best Practices for the most popular usability questionnaire Buy on Amazon | Download | |

A Practical Guide to Measuring Usability72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software Buy on Amazon | Download |

.

.

.