Measuring Usefulness

Jeff Sauro, PhD

Is it easy to use?

As important as that question is, there’s one that’s more important: Is it useful?

First and foremost, a product, website or application should solve a problem, fill a need or offer something people find useful.

In fact, people are willing to put up with poor usability if a product delivers something of great perceived value. Consider how much time you would spend learning to use software if you knew you’d have a guaranteed way to double investments in the stock market?

Conversely, it doesn’t matter how easy to use a product is if people don’t find it useful. Usefulness is the holy-grail of product design, it’s often even more important than revenue (think YouTube). How can we predict what people will use?

Predicting Usage

A good first step in predicting how much people will use a new product is to predict how how much they are using an existing product. This was one of the goals of the research done by Fred Davis in the early 1990’s[pdf]

He developed what’s come to be known as the Technology Acceptance Model (TAM) and a standardized questionnaire[pdf] that measures technology acceptance. The questionnaire consists of 2 parts, 6 items to measure usefulness and 6 items to measure ease of use.

Participants are asked to provide their level of agreement on a 7 point scale (1=likely and 7 = unlikely). Here are 12 items, first the usefulness items and then the usable items.

  1. Using [this product] in my job would enable me to accomplish tasks more quickly.
  2. Using [this product] would improve my job performance.
  3. Using [this product] in my job would increase my productivity.
  4. Using [this product] would enhance my effectiveness on the job.
  5. Using [this product] would make it easier to do my job.
  6. I would find [this product] useful in my job.
  7. Learning to operate [this product] would be easy for me.
  8. I would find it easy to get [this product] to do what I want it to do.
  9. My interaction with [this product] would be clear and understandable.
  10. I would find [this product] to be flexible to interact with.
  11. It would be easy for me to become skillful at using [this product].
  12. I would find [this product] easy to use.

Davis had current users of software (from the late 80’s and early 90’s) answer the TAM items as well as their current usage of the systems being examined (text editor and electronic mail) to see how well he could predict their usage.

Note: This article was updated in 2017 to show all 12 items from the TAM.

Usefulness is 1.5 times more important than Ease of Use

Davis found the usefulness items were able to predict around 36% of actual reported usage. The usability items had a more indirect effect on use by impacting attitudes towards usage. Davis found that answers to the usefulness scores were about 1.5 times more important than usability scores in predicting actual use.

In short, you can explain a lot of whether people will accept or reject a technology based on how useful and usable it is.

Measuring Ease of Use

As it happens, several of the 10 ease of use items Davis used are the same or very similar to those used in the System Usability Scale (SUS). So when measuring product usability it’s better to just use the SUS which has been studied refined and is widely used. For website usability use the SUPR-Q which has 4 items that effectively substitute for the SUS, are designed specifically for websites and are normalized against a database of 200 websites.

Caution

It would be great to ask people a few questions to see if they would both use and then purchase a product. Asking people what they want in a new product is notoriously unreliable–as Steve Jobs has famously noted.

We can and should be critical of data based on users’ predictions of their future behavior from focus groups, surveys or even the most complex statistical analysis. However, being critical is different than dismissing data altogether. Behavioral metrics are fickle but they’re often the only data we have and they are often quite valid predictors. I wonder what the usefulness scores of Netflix would have looked like prior to the recent maligned price and service changes?

Here’s how to use this information:

  • Consider measuring usefulness: by presenting the 10 TAM items to your users or potential users of a product
  • Measure over time: See how changes in features correspond to changes in reported use.
  • Predict Usage and adoption: Ask the TAM prior to release of a product then after users have some time with it. See how well their usage was predicted by the TAM items.
  • Generate Benchmarks: Collect enough Technology Acceptance data to create an internal set of benchmarks when prioritizing products as I’ve done with usability and Net promoter benchmarks.

 

0
    0
    Your Cart
    Your cart is emptyReturn to Shop
    Scroll to Top