Continuous Data
Efficiency is one of the cardinal aspects of a product’s usability. The amount of time it takes for a user to complete a task is one of the best predictors of efficiency because it:
- Illustrates the slightest variability (in seconds) that are more difficult to detect in other measures like errors
- Is a continuous measurement meaning it can be subdivided infinitely (unlike task completion which is discrete-binary) and many more statistical tests can be used to detect meaning in your data.
How to Measure Task Time
It’s very important to know there are many complexities when deciding to measure task times. Upon initial consideration it may seem like a straightforward measurement: Get a stop watch and start timing when the user starts the task and stop when they’re done and record the time.
When to Start and When to Stop
The first problem you will encounter is when to start the timing: Is it when the user is handed the scenario and task description? Or is it when the user first clicks on something? And when do you stop? When the user says they’re done or when they technically completed the task but before they spent time double checking their work?
There are different definitions of what constitutes task times. The most important thing is to be consistent and note your timing standard. I’ve settled on the following:
Start Time: The user’s first orientation toward the application. That is, as soon as the user finished digesting the scenario (suggesting they understand the task ) and they make some orientation toward the monitor, usually with the mouse.
End Time: When the user signals they’re done (either verbally or through some non-verbal means like writing “Done” on the task page). Other methods might include having the user turn the monitor on and off between tasks. I prefer to start timing after I think the user understands the tasks (as opposed to when they begin reading the task) similar to how they would behave on their job: they know what they need to do they just need to “do it” in the software. It’s best to minimize the confounding effects of reading and comprehension speeds.
Technically Done and “I’m Done”
I also find it helpful to note when the user technically completes the task and when they state they completed the task. You want to be able to decipher the timing data and identify the following causes for the differences between technically complete and subjectively complete:
- Hawthorne Effect: In some cases users are just being thorough because they know they’re being tested(regardless of what you tell them otherwise) so they will double-check unnaturally.
- Software isn’t reassuring enough: Some tasks don’t provide the user with enough information for them to know if they successfully completed the task. Sometimes a screen may be lacking a confirmation message.
- System Lag Time: Some”CPU” intensive tasks that take more than a couple seconds to complete. Note when the user completes the task and when the system completes the task. It’s also interesting to note what the user does during the perceived eternity while watching the hour glass.
