Usability, Customer Experience & Statistics

Quantifying the User Experience: Practical Statistics for User Research the Book (TOC)

Authors: Jeff Sauro & Jim Lewis

Jeff Sauro • December 21, 2010

This is the final outline of our book with Morgan Kaufmann. It will bring together almost a decade of research on finding the best statistical approaches to solving the most common issues in user research. Publication date is April 15 2012.

  1. Introduction & How to Use this Book
    1. Visual Guide to What Test
    2. Skipping the formulas

  2. Quantifying User Research
    1. What is User Research?
    2. Usability Tests (lab and remote)
      1. Benchmarking
      2. Comparative Testing
      3. Qualitative Studies
    3. Surveys
    4. Requirements Gathering
    5. A/B Testing
    6. Questionnaires
    7. Using Inferential Statistics with usability Data
    8. Samples Size, Normality and other statistical concerns
    9. Measuring Usability: Quantifiable Aspects of Usability
      1. Introduction: Metrics as independent to formative and summative tests
        1. Completion
        2. Time
        3. Satisfaction
        4. Errors
        5. Clicks / Page Views
        6. Combined Scores
        7. Problems Discovered

  3. How precise are our estimates: Confidence Intervals
    1. Confidence Interval = Twice the Margin of Error
    2. Confidence Intervals Provide Precision & Location
    3. Three Components of a Confidence Interval
      1. Confidence Level
      2. Variability
      3. Sample Size
    4. Confidence Interval for a Completion Rate
      1. Confidence Interval History
      2. Wald Interval: terribly inaccurate for small samples
      3. Exact Confidence Interval
      4. Adjusted-Wald: Add Two Successes & Two Failures
      1. Best Point Estimates for a Completion Rate
        1. How accurate are point estimates from small samples?
    5. Confidence Interval for a Problem Occurrence
    6. Confidence Interval for Rating Scales and other Continuous Data
    7. Confidence Interval for Task Time Data
      1. Mean or Median Task Time?
      2. The Geometric Mean
    8. Log Transforming Confidence Intervals for Task Time Data
    9. Confidence Interval for a Median

  4. Did we meet or exceed our goal? 
    1. Introduction
      1. One-Tailed and Two-Tailed Tests
    2. Comparing a Completion Rate to a Benchmark
      1. Small Sample Test
        1. Mid-Probability
      2. Large Sample Test
    3. Comparing a Satisfaction Score to a Benchmark
      1. Do at Least 75% Agree? Converting Continuous Ratings to Discrete
        1. Disadvantages to Converting Continuous Ratings to Discrete
        2. Net Promoter Score
    4. Comparing a Task Time to a Benchmark

  5. Is there a statistical difference between products?
    1. Comparing two Means (Rating Scales & Task Times)
      1. 2-sample t-test (between subjects)
        1. Confidence Interval around the Difference
      2. Paired t-test (within subjects)
        1. Confidence Interval around the Difference
    2. Comparing Completion Rates
      1. Small Samples : Fisher Exact Test
      2. Large-Samples : The N-1 2-proportion test
        1. Confidence Interval around the Difference
        2. Relationship between Chi-Square Tests and 2-proportion tests
      3. A/B Testing & Conversion Rates

  6. What Sample Sizes Do We Need?  Part 1: Summative Usability Studies
    1. Introduction
      1. Why Do We Care?
      2. The Type of Usability Study Matters
      1. Basic Principles of Summative Sample Size Estimation
    2. Estimating Values
      1. Example 1: A Realistic Usability Testing Example Given Estimate of Variability
      2. Example 2: An Unrealistic Usability Testing Example
      3. Example 3: No Estimate of Variability
    3. Comparing Values
      1. Example 4: Comparison with a Benchmark
      2. Example 5: Within-Subjects Comparison of an Alternative
      3. Example 6: Between-Subjects Comparison of an Alternative
      4. Example 7: Where's the Power?
    4. What Can I Do to Control Variability
      1. Sample Size Estimation for Binomial Confidence Intervals
      2. Binomial Sample Size Estimation for Small Samples
      3. Sample Size for Comparison with a Benchmark Proportion
      4. Sample Size Estimation for Proportions & Chi-Squared Tests

  7. What Sample Sizes Do We Need?  Part 2 :  Problem Discovery
    1. Using a Probabilistic Model of Problem Discovery to Estimate Sample Sizes for Formative User Research
    2. The famous equation (P(x ≥ 1) = 1 (1 p)n
      1. Deriving a sample size estimation equation from 1 (1 p)n
      2. Using the tables to plan sample sizes for formative user research
    3. Assumptions of the Binomial Probability Model
    4. Additional Applications of the Model
      1. Estimating the composite value of p for multiple problems or other events
      2. Adjusting small-sample composite estimates of p
    5. Estimating p
      1. Adjusting the Initial Estimate of p
      2. Using the Adjusted Estimate of p
      3. Investigating Sample Size Effectiveness
      4. Estimating the Number of Problems Available for Discovery
      5. What Affects the Value of p?
  8. Attitudinal Measurement with Questionnaires
    1. Scales, Labels and Points
    2. Post-Task Questionnaires
    3. ASQ, SMEQ, 1-question Likert
    4. Post-Test
    5. SUS, SUMI, PSSUQ, Homegrown scales
    6. Usability and Loyalty
    7. Net Promoter Scores and SUS

  9. Controversies in Measurement & Statistics
    1. Industrial versus Scientific: Purpose of statistics is to help in better decision making over the long run
    2. Multi-Point Scales
    3. p-values and NHST
    4. Parametric versus Non-Parametric Statistics
    5. Which confidence level
    6. When x=n or x=0 what confidence level do you use?
    7. Multiple testing versus omnibus testing
    8. 2 x 2 tables

  10. Final Thoughts on Statistics for User Research

  11. Appendix A:  A Crash Course in Fundamental Statistical Concepts
    1. Central Tendency: Mean & Median
    2. Standard Deviation & Variance
    3. Population Parameters and Sample Statistics
    4. Standard Deviation
    5. Margin of Error
    6. Alpha
    7. Standard Error of the Mean
    8. Central Limit Theorem
    9. The normal distribution
    10. The Binomial Distribution
    11. Normal Approximation to the Binomial
    12. Introduction to Hypothesis Testing
      1. The Null and Alternative Hypothesis (Ho and Ha)
      2. Type I and Type II Errors
      3. Confidence and Power
      4. Making decisions from p-values
        1. If p is low reject the Ho
      5. One and Two Tailed Tests
    13. Mechanics of Test Statistics
      1. z statistics
      2. t-statistics

About Jeff Sauro

Jeff Sauro is the founding principal of MeasuringU, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 5 books on statistics and the user-experience.
More about Jeff...

Learn More

UX Bootcamp: Denver: Aug 17-21, 2016

Related Topics

Statistics, UX, User Research

Posted Comments

There are 6 Comments

November 1, 2011 | Muhammad Ali wrote:

Can't wait to buy a copy! Make it an ebook and skip the paper, shorten the time to print! and aso
it is difficult for me to buy this book. If you send me free copy o this book by an email i will
be very thankful to you 

February 8, 2011 | Raphal Yharrassarry wrote:

2012 ! not before ? In 2012, it's the end of the world... ;-) 

January 10, 2011 | Jeff Sauro wrote:

It'll be available from Amazon early next year and available from my website.

The closest thing I have available now to this book is How to Conduct a Quantitative Usability Test which can be downloaded electronically or purchased in hard-copy from Amazon (marketed there as A Practical Guide to Measuring Usability

It contains a couple questions on within versus between subjects tests in usability. 

January 10, 2011 | Dennis wrote:

where can i find and buy the book ? (arrived at the page via "within subjects test usability" query on google, while researching for my CUA exam from HFI) 

December 25, 2010 | John Romadka wrote:

Can't wait to buy a copy! Make it an ebook and skip the paper, shorten the time to print! 

December 21, 2010 | Susan Weinschenk wrote:

Too long to wait! Seriously, I look forward to buying a copy for myself and a few more for friends and colleagues. Looks like a great book! 

Post a Comment


Your Name:

Your Email Address:


To prevent comment spam, please answer the following :
What is 5 + 3: (enter the number)

Newsletter Sign Up

Receive bi-weekly updates.
[6128 Subscribers]

Connect With Us

UX Bootcamp

3 Days of Hands-On Training on UX Methods, Metrics and Analysis
Denver: Aug. 17-19 2016

Our Supporters

Use Card Sorting to improve your IA

Userzoom: Unmoderated Usability Testing, Tools and Analysis

Loop11 Online Usabilty Testing


Jeff's Books

Customer Analytics for DummiesCustomer Analytics for Dummies

A guidebook for measuring the customer experience

Buy on Amazon

Quantifying the User Experience: Practical Statistics for User ResearchQuantifying the User Experience: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to Quantifying the User ExperienceExcel & R Companion to Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download