Jeff Sauro • April 17, 2012

One of the simplest ways to measure any event is a binary metric coded as a 1 or 0.Such a metric represents the presence or absence of just about anything of interest: Yes/ No, Pass/ Fail, Purchase/No Purchase, On/Off.

Fundamentally the binary system is at the heart of computing as we know it. It also plays a critical role in user research.

Binary measures can even apply to traditionally qualitative research such as noting whether a user mentions distrust of sales people or marketing materials in an interview.

By adding up all the 1's and dividing by the total number observed you get a proportion. So if 5 out of 10 users mention distrusting marketing materials in an interview you get a proportion of 5/10 = .5 which is often expressed as a percentage 50%.

You can use proportions to help make data driven decisions just about anywhere: Which design converts more? Which product is preferred? Does the new interface have a higher completion rate? What proportion of users had a problem registering?

When it comes to comparing independent proportions you'll need to conduct a statistical test called the 2 proportion test (which is equivalent to the Chi-Square test).

We talk a lot about comparing two proportions in our book Quantifying the User Experience (Chapter 5) and recommend a slight adjustment to the typical formulas you might be familiar with so it works for small and large sample sizes.

To make it easier to compare proportions, I've created a simple online web-calculator which will do the statistical calculations for you. Just enter two proportions to see if the difference between them is more likely due to chance or more likely a legitimate difference.

Here are 6 examples of proportions and the statistical results to get you thinking:

- Completion Rates: If 11 out 12 users complete a task on Design A and only 5 out of 10 can complete the same task on Design B, then we can be 97% confident more users can complete the task on Design A.
- Conversion Rates: A large blue button was shown to 455 users and 37 (8%) purchased a product. A large red button was shown to 438 different users and 22 purchased the product (5%). There is a 94% chance the blue button will sell more products.
- Problem Occurrence: 4 out of 7 users received at least one error message when entering alerts and notifications into their profile on a credit card website. After a redesign, 1 out of 7 had at least one error. There is an 89% chance the number of errors has been reduced when setting account alerts.
- Proportion Recommending: 89 out of 100 (93%) customers said they recommended Smart Phone A to a friend in the last year compared to 67 out of 93 (72%) for Smart Phone B. There is a 99.7% chance this retroactive recommend rate is different.
- Proportion detracting: Prior to the change in the return policy, 49 out of 100 (49%) customers surveyed were detractors. After the change in policy 40 out of 96 (42%) were. There is about a 69% chance the difference is not due to chance (good, but not overwhelming evidence).
- Proportion that completed a task in less than 30 seconds: 4 out of 9 users could add a new contact in CRM application A in less than 30 seconds. Eleven out of 12 could on CRM application B. There is a 97% chance if we tested all users, more would complete the tasks on App B.

5 Visualization Techniques for Managing the Customer Experience

10 Things to Know about a Key Driver Analysis

10 Things to Know about Usability Problems

Should you use 5 or 7 point scales?

5 Examples of Quantifying Qualitative Data

Confidence Interval Calculator for a Completion Rate

A Brief History of the Magic Number 5 in Usability Testing

Why you only need to test with five users (explained)

What five users can tell you that 5000 cannot

Does better usability increase customer loyalty?

The Five Most Influential Papers in Usability

8 Ways to Show Design Changes Improved the User Experience

97 Things to Know about Usability

Nine misconceptions about statistics and usability

.

Customer Analytics for DummiesA guidebook for measuring the customer experience Buy on Amazon | |

Quantifying the User Experience 2nd Ed.: Practical Statistics for User ResearchThe most comprehensive statistical resource for UX Professionals Buy on Amazon | |

Excel & R Companion to Quantifying the User ExperienceDetailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R Buy on Amazon | Download | |

A Practical Guide to the System Usability ScaleBackground, Benchmarks & Best Practices for the most popular usability questionnaire Buy on Amazon | Download | |

A Practical Guide to Measuring Usability72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software Buy on Amazon | Download |

.

.

.