Usability, Customer Experience & Statistics

If you could only ask one question, use this one.

Jeff Sauro • March 2, 2010

Was a task difficult or easy to complete? Performance metrics are important to collect when improving usability but perception matters just as much. Asking a user to respond to a questionnaire immediately after attempting a task provides a simple and reliable way of measuring task-performance satisfaction. Questionnaires administered at the end of a test such as SUS, measure perception satisfaction.

There are numerous questionnaires to gather post-task responses, some of the more popular ones are :
  1. ASQ PDF: After Scenario Questionnaire (3 Questions) 
  2. NASA-TLX : NASA's task load index is a measure of mental effort (5 Questions) 
  3. SMEQPDF: Subjective Mental Effort Questionnaire (1- Question)
  4. UMEPDF : Usability Magnitude Estimation  (1-Question)
  5. SEQ PDF: Single Ease Question

The SEQ is a new addition which Joe Dumas and I tested two years ago PDF and found it to perform very well.  In addition to the usual psychometric properties of being reliable, sensitive and valid, a good questionnaire should also be:
  1. short
  2. easy to respond to
  3. easy to administer
  4. easy to score

The SEQ is all four. You can administer it on paper, electronically on any web survey service or even verbally.

Overall, this task was?

Very Difficult           Very Easy

Figure 1: The Single Ease Question (SEQ).


For the past year I've been collecting data using the SEQ on numerous tasks on websites and applications. My goal is to assemble a large database of tasks to generate a standardized post-task SEQ score. It would be nice to have standardized task completion rates and task times for classes of tasks but I've found slight differences in tasks scenarios make aggregating data difficult.

The beauty of the SEQ is that users build their expectations into their response. So while adding or removing a step from a task scenario would affect times, users adjust their expectations based on the number of steps and respond to the task difficulty accordingly. On your next task use the SEQ.




About Jeff Sauro

Jeff Sauro is the founding principal of MeasuringU, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 5 books on statistics and the user-experience.
More about Jeff...


Learn More

Related Topics

Satisfaction, SEQ, Questionnaires
.

Posted Comments

There are 11 Comments

December 5, 2015 | J. Sigurdson wrote:

It is great that you are / were trying to establish base rates ...rnrnHowever, sorry I am following the logic of your above SEQ. rnI don't see how it accounts for subjectivity / idiocyncratic biases of respondents. rnrnAlso one only gets a fixed snapshot respondents' self-rating which has a limited shelf life. A task /skill e.g. backhand shot in squash is initially difficult for most players, but with a few months of practice is no longer seen or actually difficult, which again varies.rnrnWithout a few baseline questions to anchor respondents' internal scale, I don't see how one can reasonable compare or aggregate ... subjective values. It might be necessary to have some time gap in baseline testing before the actual questionnaire to minimize the chance of priming / salience influencing the results.rnrnBe well,rn JS 


December 2, 2015 | Navatha wrote:

Good article with excellent idea! I appreciate your post.Regards<a href=http://sarkarinaukrizone.com/sarkari-result/ > Sarkari Result </a> 


June 10, 2015 | M Turner wrote:

Hi Jeff, the ASQ link needs updating; it takes me to this page: http://drjim.0catch.com/ rather than the PDF doc. 


October 7, 2012 | Paul Smith wrote:

I agree with this as a most useful question to help define measurable ux metrics. However, I don't like your example of 'very difficult' then 5 unlabeled choices and then 'very easy'. You're going to get arbitrary results on people who pick radio 4 & 5 etc. I prefer scoring it this way, in buckets that so far work for most test subjects I've reviewed: 0 - unable to perform (within a minute or two), 1 - very difficult, 2 - difficult 3 - okay (acceptable, not too difficult), 4 - easy, 5 - very easy 


February 18, 2012 | Jeff Sauro wrote:

@Ryan,

Good questions. You know, the original version of that scale had no labels--it came from some studies from Tullis and Stetson. Since the original publication of this blog and the CHI paper I HAVE labeled my radio buttons from 1 to 7 with the labels over the 1 and 7 only. I agree with your sentiment on giving users that number. I'm guessing there would be a modest difference between the label and unlabelled version but I don't have data on that. Given what I've seen, I'm inclined to lean in the favor of easier to respond to for the user unless I find data to contradict my intuition.

I have been using the SEQ for quite a while now and have built up a normed database that I'll be releasing as a downloadable product--it will convert raw scores to percentile ranks. In the absence of that data for now, using 5.6 (or something close) is a good idea.

Let me know if you find anything different.  


February 17, 2012 | Ryan Knutzen wrote:

Your SEQ question is a 7-point rating scale, but the points are not numbered. Is that intentional? Do you know if it has an impact on the ratings they give? I prefer to number each point as well as label the endpoints, to help users scan and select the right response (7 radio buttons always feels a little dizzying at first glance).
Also, similar to Timo's last question, do you have an "average" score for this SEQ question? Do you think the 5.6 average from Nielsen's 1994 article applies in this case?
 


November 1, 2011 | sahil wrote:

Why railways having sleeper on ralway track.........................................................................................?rn 


April 7, 2010 | Timo Jokela wrote:

David: a bit delayed comment. My setting is creating usability requirements into a RFP. I would like to set a verifiable usability requirements as acceptance criteria for the system to be developed. 


March 18, 2010 | David Bishop wrote:

Timo, Jeff, we've found that in many cases, it's more important to show that usability is improving than to have a specific numerical requirement for 'good.' Pessimists could chalk this up to state of the art (usability is generally poor enough that any raising of the bar will be of value). Optimists can point out that an organization that measuring usability and aiming for improvement is an organization that has a mature process ( la CMM), which may be more important that setting a specific target. 


March 13, 2010 | Jeff Sauro wrote:

Timo, Great question. Rating scale data is only as good as what you can compare the results to. With the SEQ I have used it for over 100 tasks so my raw scores can be expressed as a percentile rank. Later this year I'll talk more about that database so others can generate standardized ease scores by task. 


March 11, 2010 | Timo Jokela wrote:

I would be most interested to use a question such as SEQ for usability requirements.

But what would be the value for 'good' usability? Is 7 good enough? Or 8?

How would you comment on this? 


Newsletter Sign Up

Receive bi-weekly updates.
[6505 Subscribers]

Connect With Us

Our Supporters

Use Card Sorting to improve your IA

Loop11 Online Usabilty Testing

Usertesting.com

Userzoom: Unmoderated Usability Testing, Tools and Analysis

.

Jeff's Books

Customer Analytics for DummiesCustomer Analytics for Dummies

A guidebook for measuring the customer experience

Buy on Amazon

Quantifying the User Experience 2nd Ed.: Practical Statistics for User ResearchQuantifying the User Experience 2nd Ed.: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to the 2nd Ed. of Quantifying the User ExperienceExcel & R Companion to the 2nd Ed. of Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download

.
.
.