Usability, Customer Experience & Statistics

10 Things to Know about Task Times

Jeff Sauro • August 9, 2011

Time is a metric we all understand so it's no wonder it's one of the core usability metrics.

Perhaps it's something about the precision of minutes and seconds that demands greater scrutiny. 

There's a lot to consider when measuring and analyzing task time. Here are 10 of them.

  1. Task times are collected in about half of formative usability tests and 75% of summative tests.

  2. Task times can be great for diagnosing usability problems. A long task time is often caused by problems with the interaction with the interface.

  3. Time on task can be collected even when users think aloud. Some studies show that it actually increases users' speed, rather than decrease it.  This is different than probing a user (asking them questions during the task). Retrospectively probing users after the task allows you to both collect more stable task times and better understand any problems they were having.

  4. There are three core ways to report task-times:
    1. Average Task Completion Time: Include only users who completed the task successfully. This is the most common one to report.
    2. Mean Time to Failure: The average time users are spending on the task before they give up or complete the task incorrectly.
    3. Average Time on Task: The total duration users are spending on your task.

  5. When reporting the average task time, use the geometric mean for small samples (< 25) and the median for larger samples. The arithmetic mean tends to be heavily skewed by a few outliers and the median tends to be less accurate for small sample sizes.

  6. Compute a confidence interval around the task time after log-transforming your data. You can use this free calculator or download the Excel version.

  7. Time saved can be an excellent measure of productivity: Showing that a new design can substantially reduce the time it takes users to complete a task is a metric even executives understand.

  8. You can estimate the task time for experienced users who commit no errors using Keystroke Level Modeling.  You can consider this the fastest time to complete a task.

  9. How long should a task take?  There are several strategies for identifying an acceptable task time.
    1. Find task times for competing products or products with acclaimed user interfaces.
    2. Find the time for the prior version or similar version of the product.
    3. Find the times from the most satisfied users. I call this a bootstrapped specification limit[pdf].
    4. Carrying out the task without use of a computer system (or on an older platform e.g.,  in DOS)
    5. Identifying the expert or fastest task time and setting the unacceptable condition to 1.5 times (or another multiple) this time for each task. You can use the results from a KLM analysis. Note: It's unclear what a good multiple should be. More research is needed to make this approach more meaningful so use it as a last resort.

  10. Task Times are Relative: Don't get hung on what the "right" task time is. All task times will be wrong. Lab based studies are too idealized. Unmoderated remote times are easily affected by users being distracted by other activities.  It's the relative comparison using the same method that provides the most meaning. 
For more information on collecting and analyzing tasks times see How to Conduct a Quantitative Usability Test, also available in print from Amazon under the title a Practical Guide to Measuring Usability.

About Jeff Sauro

Jeff Sauro is the founding principal of MeasuringU, a company providing statistics and usability consulting to Fortune 1000 companies.
He is the author of over 20 journal articles and 5 books on statistics and the user-experience.
More about Jeff...

Learn More

You Might Also Be Interested In:

Related Topics

Task Time

Posted Comments

There are 4 Comments

February 12, 2013 | Mari-Carmen wrote:

HirnVery useful blog and books, thanks.rnrnI have a question about how to measure task completion time with this situation:rnrn10 users performed a task with the interface "A" and 10 different users performed the same task with the interface "B". 3 experts performed the task in both interfaces to have a reference time.rnrnI would like to know if there is a statistic significant difference between time people needed in interface A and B.rnWhich test should I use?rnrnHere you are the data (in seconds) for task 1 using interfaces A and B:rnrnUsers in interface A:rn23rn34rn34rn135rn14rn159rn75rn60rn19rn25rnrnUsers in interface B:rn95rn30rn25rn12rn5rn8rn22rn18rn15rn10rnrnExperts' time in A: rn10rn12rn14rn16rnrnExperts' time in B:rn12rn14rn14rn10 

July 24, 2012 | Odawe Muzan-Ekpelu wrote:

Is there a scientific (theoretical) backing to the statement 'A long task time is often caused by problems with the interaction with the interface': found in number 2 of 10 Things To Know About Task Times? 

August 11, 2011 | Petra Quilitz wrote:

For users of an interface, savings in time feel like simplicity.  

August 10, 2011 | Philip Hodgson wrote:

Great post. Thanks. Measuring time can be tricky, especially if trying to arrive at an absolute measure. Re. your point #10, one useful approach is to express the user's time relative to the time taken by an expert. Simply have 2-3 expert users complete all of the test tasks and record the mean time. Then use this as the reference time. This also lets you easily express time as a percentage (if, for example, you are trying to combine effectiveness and efficiency measures into a single metric). If the expert's time is 2 minutes, and the user's time is 4 minutes, the task gets a 50% score. This can also be used as an indication of learning still required in order to achieve expert performance. 

Post a Comment


Your Name:

Your Email Address:


To prevent comment spam, please answer the following :
What is 3 + 2: (enter the number)

Newsletter Sign Up

Receive bi-weekly updates.
[6389 Subscribers]

Connect With Us

Our Supporters

Use Card Sorting to improve your IA

Loop11 Online Usabilty Testing

Userzoom: Unmoderated Usability Testing, Tools and Analysis


Jeff's Books

Customer Analytics for DummiesCustomer Analytics for Dummies

A guidebook for measuring the customer experience

Buy on Amazon

Quantifying the User Experience 2nd Ed.: Practical Statistics for User ResearchQuantifying the User Experience 2nd Ed.: Practical Statistics for User Research

The most comprehensive statistical resource for UX Professionals

Buy on Amazon

Excel & R Companion to Quantifying the User ExperienceExcel & R Companion to Quantifying the User Experience

Detailed Steps to Solve over 100 Examples and Exercises in the Excel Calculator and R

Buy on Amazon | Download

A Practical Guide to the System Usability ScaleA Practical Guide to the System Usability Scale

Background, Benchmarks & Best Practices for the most popular usability questionnaire

Buy on Amazon | Download

A Practical Guide to Measuring UsabilityA Practical Guide to Measuring Usability

72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Buy on Amazon | Download