Usability, Customer Experience & Statistics

Blogs & Articles

Standardized questionnaires are generally more reliable and valid than homegrown ones. Many also offer the ability to convert raw scores into percentile ranks. Sometimes you may feel a previously validated questionnaire needs a change (e.g. item wording, scale labels). But before you make changes, consider whether you are measuring more than one thing and whether the existing data you get is good enough. If you decide to make changes, test the original and new items with a set of participants and examine the correlation.

Can You Change A Standardized Questionnaire?

Jeff Sauro • August 30, 2016

Standardized questionnaires are generally more reliable and valid than homegrown ones. Many also offer the ability to convert raw scores into percentile ranks. Sometimes you may feel a previously validated questionnaire needs a change (e.g. item wording, scale labels). But before you make changes, consider whether you are measuring more than one thing and whether the existing data you get is good enough. If you decide to make changes, test the original and new items with a set of participants and examine the correlation.[Read More]

Knowing how many participants you should use is your UX research study is not a simple question. What

Getting Started Finding the Right Sample Size

Jeff Sauro • August 23, 2016

Knowing how many participants you should use is your UX research study is not a simple question. What's particularly difficult about learning how to compute the right sample size is that books and articles can get overly technical; it's hard to know whether the advice is relevant to an applied research setting. In general, it can be hard to know where to start. This article includes five steps to help you start computing the right sample size for your next study.[Read More]

Contextual inquiries involve observation and semi-structured interviews to understand how participants use a product in a particular context. They are one of a number of methods used to help identify unmet user needs. They provide insight into: problems people are trying to solve, points of friction and how people go about completing tasks to accomplish goals. This article includes some background and 8 tips for making contextual inquires more successful.

The Essentials of a Contextual Inquiry

Jeff Sauro • August 16, 2016

Contextual inquiries involve observation and semi-structured interviews to understand how participants use a product in a particular context. They are one of a number of methods used to help identify unmet user needs. They provide insight into: problems people are trying to solve, points of friction and how people go about completing tasks to accomplish goals. This article includes some background and 8 tips for making contextual inquires more successful.[Read More]

If people encounter problems in the online checkout process there

Does Improving Website Navigation Pay Off?

Jeff Sauro • August 8, 2016

If people encounter problems in the online checkout process there's a clear link to conversion rates and the bottom line. But if users can't even find what they are looking for, the best checkout process won't matter. The cost of poor findability is high but does improving it really pay off? Here are five points backed by research to emphasize the importance of measuring and improving website findability.[Read More]

It

13 Things to Consider When Offering Survey Incentives

Jeff Sauro • August 1, 2016

It's no fun planning a party no one comes to. The same can be said about surveys that hardly any one takes. In the never-ending quest to collect data from customers through branded or unbranded surveys, we inevitably encounter the question: should we offer some type of incentive for people to take the survey? Here are 13 things to consider when deciding whether an incentive is the way to go.[Read More]

While it

Controlling for Brand Attitudes in UX Studies

Jeff Sauro • July 26, 2016

While it's realistic to have a mix of brand lovers and haters in a UX study, you'll often want to understand how much the experience differs despite these variations in participants' brand attitudes. Brand favorability has a strong and statistically significant impact on UX metrics. The correlation is strong enough that it often masks the differences between interfaces being compared. To control for the effects of brand favorability you can use an ANCOVA (preferred) or create a segment of low and high brand favorability participants to see whether the patterns in metrics hold.[Read More]

The mobile user experience is no longer a niche industry. It

15 Mobile UX Facts and Insights for 2016

Jeff Sauro • July 19, 2016

The mobile user experience is no longer a niche industry. It's hard to talk about the user experience of websites and software and not consider the impact and experience of mobile devices. We've presented a number of facts and insights in 2013 and 2015. Here's an updated and expanded version culled from many sources on the internet and our own mobile usability studies.[Read More]

The System Usability Scale (SUS) is the most used questionnaire to measure perceptions of usability. The SUS

5 Ways to Use the System Usability Scale (SUS)

Jeff Sauro • July 12, 2016

The System Usability Scale (SUS) is the most used questionnaire to measure perceptions of usability. The SUS's wide adoption and good psychometric properties means it's a good instrument to start measuring with. Here are five uses of the System Usability Scale: After a usability test, in a survey, with prototypes, for focused functionality, on mobile apps and more.[Read More]

Statistics can be daunting, especially for UX professionals who aren

5 Steps for Getting Started with Statistics for Research

Jeff Sauro • July 5, 2016

Statistics can be daunting, especially for UX professionals who aren't particularly excited about the idea of using numbers to improve designs. But like any skill that can be learned, it takes some time to understand statistical concepts and put them into practice. Here's five steps for getting started: understanding data types, sampling error, confidence intervals, statistical significance and knowing what test to use.[Read More]

Excel is an invaluable tool for analyzing and displaying data. In Part 1 I covered some essentials Excel skills, such as conditionals, absolute references and the fill handle. In this second part I

Essential Excel Skills For Researchers Part 2

Jeff Sauro • June 28, 2016

Excel is an invaluable tool for analyzing and displaying data. In Part 1 I covered some essentials Excel skills, such as conditionals, absolute references and the fill handle. In this second part I'll cover a few more shortcuts and advanced functionalities that mimic database manipulations (without needing to know SQL).[Read More]

Excel is a powerful program. It

Essential Excel Skills For Researchers Part 1

Jeff Sauro • June 21, 2016

Excel is a powerful program. It's like an onion, peeling back layers to reveal increasingly specialized functions. It can take years to master. But after you get beyond the basic idea of the spreadsheet and simple formulas, you should master the following functions and techniques that make data analysis more efficient and powerful. Part 1 of 2.[Read More]

A lot goes into a benchmarking study, and any of those things can go wrong. It

5 Common Mistakes in UX Benchmark Studies

Jeff Sauro • June 14, 2016

A lot goes into a benchmarking study, and any of those things can go wrong. It's important to get the details right because you'll use the benchmarks to make comparisons over time and the wrong decisions can have an impact that last for years.Here are five of the more common mistakes made when conducting benchmark studies and what you can do to prevent them.[Read More]

While it

How to Assign the Severity of Usability Problems

Jeff Sauro • June 7, 2016

While it's generally straightforward to count how many times you observe a problem in a usability test, assigning severity ratings to problems can be more challenging. There are often too many problems to fix. Severity ratings help development teams prioritize what does and doesn't get fixed. Here are five steps to follow for establishing a system for assigning problem severity after observing problems in a usability test: Agree on a severity scale; Train evaluators; Have at least two evaluators rate independently; Evaluate agreement; Average or reconcile the differences in ratings.[Read More]

While it

Should All Graphs Start at 0?

Jeff Sauro • May 31, 2016

While it's a good idea to have best practices with displaying data in graphs, the "show the zero" is a rule that clearly can be broken. But showing or not showing the zero alone is not sufficient to declare a graph objective or conversely "deceptive." There isn't an objective graph. But if people want to deceive, they will and the unfocused reader may certainly be swayed. But often displaying the data with or without the zero can provide a different perspective for readers to reconsider the context and the implications.[Read More]

Yes, of course you can. But it depends on who you ask! It

Can You Take the Mean of Ordinal Data?

Jeff Sauro • May 24, 2016

Yes, of course you can. But it depends on who you ask! It's a common question and point of contention when measuring human behavior using multi-point rating scales. Whether someone tells you it's permissible to take the average of ordinal data depends on their view of measurement theory—and not all people agree. Using the mean of ordinal data is fine; just be careful not to make interval or ratio statements about your data -- even researchers who take a more relaxed view of averaging ordinal data would disagree with that practice.[Read More]

The successful researcher, regardless of title, should understand how to combine traditional market research and UX research activities for the best results. If you

Combining UX Research with Market Research

Jeff Sauro • May 17, 2016

The successful researcher, regardless of title, should understand how to combine traditional market research and UX research activities for the best results. If you're in either role, you should understand the tools and techniques that help define what customers think and what they do—and that means blending methods and mindsets.[Read More]

When creating a survey, one important variable to consider is whether to brand the survey as coming from your organization or having a third-party research firm host and send an anonymous survey. While a branded survey is likely to increase response rates, it comes with some drawbacks too.

The Pros and Cons of a Branded Survey

Jeff Sauro • May 10, 2016

When creating a survey, one important variable to consider is whether to brand the survey as coming from your organization or having a third-party research firm host and send an anonymous survey. While a branded survey is likely to increase response rates, it comes with some drawbacks too.[Read More]

Graphs of difference scores are a helpful visualization technique to help highlight differences compared to displaying raw scores. Their main advantage can also be a disadvantage as even small differences can look important. In customer research and analysis in general, while the graph maker can help, it

Visualizing Data: Raw vs Difference Scores

Jeff Sauro • May 3, 2016

Graphs of difference scores are a helpful visualization technique to help highlight differences compared to displaying raw scores. Their main advantage can also be a disadvantage as even small differences can look important. In customer research and analysis in general, while the graph maker can help, it's ultimately up to the viewer to understand if differences depicted have an impact that's meaningful or more modest.[Read More]

Whether you

Create a UX Measurement Plan

Jeff Sauro • April 26, 2016

Whether you're introducing how to measure user experience to an organization or trying to advance the maturity of your UX practice, you need a plan for measuring and improving the user experience. With a good idea about who your users are and how to collect data from them, here's a high-level plan to start measuring and then improving the user experience in 9 steps.[Read More]

Benchmarking is part of maintaining a healthy user experience for your website, app or product. Here is a list of common points to discuss and decide when embarking on a benchmarking project.

A Checklist For Planning a UX Benchmark Study

Jeff Sauro • April 19, 2016

Benchmarking is part of maintaining a healthy user experience for your website, app or product. Here is a list of common points to discuss and decide when embarking on a benchmarking project.[Read More]

All too often efforts get stymied in a quest for perfect data, the perfect metric, or the perfect method—what a lot of people call planning paralysis. Don

Better to be Approximately Right than Exactly Wrong

Jeff Sauro • April 12, 2016

All too often efforts get stymied in a quest for perfect data, the perfect metric, or the perfect method—what a lot of people call planning paralysis. Don't let a quest for perfect data prevent you from collecting any data! Look for sound approximations that get you to a "good enough" place that accomplishes the job and answers your research questions.[Read More]

It

5 Ways to Find Out More About Your Customers

Jeff Sauro • April 5, 2016

It's fundamental to creating both a usable customer experience and a better business: you need to know who your customers are. It however can be surprisingly difficult for organizations to connect with their customers to collect information. This article describes five methods (each with their strengths and weaknesses) for collecting key demographic and psychographic information about your customers.[Read More]

Your job title doesn

6 Best Practices for Using Numbers to Inform Design

Jeff Sauro • March 22, 2016

Your job title doesn't have to be "researcher" or "statistician" to use data to drive design decisions.You can apply some best practices even when numbers aren't your best friend. Here are six best practices for using numbers to inform your design efforts that don't require a career change or advanced degree in math. [Read More]

As if the Net Promoter Score didn

Is the Net Promoter Score a Percentage?

Jeff Sauro • March 15, 2016

As if the Net Promoter Score didn't already stir up enough strong opinions about whether it's the "right" metric for organizations, now there's a new controversy: how to display it. Is it an NPS of 25% or 25? Do you add or omit the percentage sign? Here are my thoughts on this % tempest in a teapot.[Read More]

To loosely quote Lord Kelvin, when we can measure something and express it in numbers, we understand and manage it better. Measuring usability allows us to better understand how changes in usability affect customer satisfaction and loyalty. Usability can and should be measured on mobile apps, enterprise accounting software, early stage prototypes, or mature websites. While devices and users will differ, here are ten core concepts to understand when measuring usability that are likely to remain constant.

10 Essentials of Measuring Usability

Jeff Sauro • March 8, 2016

To loosely quote Lord Kelvin, when we can measure something and express it in numbers, we understand and manage it better. Measuring usability allows us to better understand how changes in usability affect customer satisfaction and loyalty. Usability can and should be measured on mobile apps, enterprise accounting software, early stage prototypes, or mature websites. While devices and users will differ, here are ten core concepts to understand when measuring usability that are likely to remain constant.[Read More]

With the proliferation of big data, the number of statistical tests we can perform seems endless. But the number of fluke discoveries we

How to Handle Multiple Comparisons

Jeff Sauro • March 1, 2016

With the proliferation of big data, the number of statistical tests we can perform seems endless. But the number of fluke discoveries we're likely to detect has increased as well. One of the better known methods for managing this false positive rate is the Bonferroni correction, however, it tends to be too conservative and introduces too many false negatives. A better approach that balances both false positives and false negatives is the Benjamini-Hochberg method which is explained with examples in this article.[Read More]

In many cases, the judgment from multiple people collected independently and then aggregated, is better than even the best individual judgment. The idea of aggregating results is a powerful methodological tool that can smooth out unusual forecasts, scientific conclusions, and judgments from experts and novices alike. It

The Benefits of Aggregating Judgment

Jeff Sauro • February 23, 2016

In many cases, the judgment from multiple people collected independently and then aggregated, is better than even the best individual judgment. The idea of aggregating results is a powerful methodological tool that can smooth out unusual forecasts, scientific conclusions, and judgments from experts and novices alike. It's the power behind meta-analysis and using the average of several polls to predict the winner of an election. It can also be applied to user and customer research as well.[Read More]

For most customer research, you

5 Steps for Better Customer Sampling

Jeff Sauro • February 16, 2016

For most customer research, you're rarely able to measure the attitudes or behaviors of everyone. Instead you take a sample of your customers and use this sample to make inferences about the rest of your customers. While sampling is efficient and statistically sound, it comes with some risks. Here are five steps to help reduce some of the risks and make sampling your customer more effective.[Read More]

Expert reviews aren

5 Steps to Conducting an Effective Expert Review

Jeff Sauro • February 9, 2016

Expert reviews aren't' a substitute for usability testing and don't provide metrics for benchmarking. But they are an effective and relatively inexpensive way to uncover the more obvious pain points in the user experience. Expert reviews are best used when you can't conduct a usability test or in conjunction with insights collected from observing even just a handful of users attempting realistic tasks on a website or application. The following five steps for conducting an effective expert review aren't going to make you an "expert" in interface evaluation immediately, but if you apply them with enough practice eventually they might![Read More]

False positives are a fact of life when trying to separate the signal from the noise in UX research. As the amount of data we use to make decisions increases, the reality of dealing with false positives does too. Two common types of false positives are phantom usability issues and illusory differences.

While we can never completely eliminate false positives, we can minimize them. In UX research this is best done by managing the false positive rate, replicating studies, and triangulating data with complementary methods and multiple evaluators.

Managing False Positives in UX Research

Jeff Sauro • February 2, 2016

False positives are a fact of life when trying to separate the signal from the noise in UX research. As the amount of data we use to make decisions increases, the reality of dealing with false positives does too. Two common types of false positives are phantom usability issues and illusory differences. While we can never completely eliminate false positives, we can minimize them. In UX research this is best done by managing the false positive rate, replicating studies, and triangulating data with complementary methods and multiple evaluators.[Read More]

Showing the latest 30 posts. Show All 371 Blog Posts

Newsletter Sign Up

Receive bi-weekly updates.
[6222 Subscribers]

Connect With Us

Our Supporters

Usertesting.com

Loop11 Online Usabilty Testing

Use Card Sorting to improve your IA

Userzoom: Unmoderated Usability Testing, Tools and Analysis

About Jeff Sauro

Jeff Sauro is the founding principal of MeasuringU, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 5 books on statistics and the user-experience.
More about Jeff...

.

Jeff's Books

Customer Analytics for DummiesCustomer Analytics for Dummies

A guidebook for measuring the customer experience

Buy on Amazon

Quantifying the User Experience: Practical Statistics for User ResearchQuantifying the User Experience: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to Quantifying the User ExperienceExcel & R Companion to Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download

.
.
.