Measuring UsefulnessThe Technology Acceptance Model (TAM)
Jeff Sauro • November 1, 2011
Is it easy to use?
As important as that question is, there's one that's more important: Is it useful?
First and foremost, a product, website or application should solve a problem, fill a need or offer something people find useful.
In fact, people are willing to put up with poor usability if a product delivers something of great perceived value. Consider how much time you would spend learning to use software if you knew you'd have a guaranteed way to double investments in the stock market?
Conversely, it doesn't matter how easy to use a product is if people don't find it useful. Usefulness is the holy-grail of product design, it's often even more important than revenue (think YouTube). How can we predict what people will use?
A good first step in predicting how much people will use a new
product is to predict how how much they are using an existing product. This was one of the goals of the research done by Fred Davis in the early 1990's[pdf]
He developed what's come to be known as the Technology Acceptance Model (TAM)
and a standardized questionnaire[pdf]
that measures technology acceptance. The questionnaire consists of 2 parts, 10 items to measure usefulness and 10 items to measure ease of use.
Participants are asked to provide their level of agreement on a 7 point scale (1=strongly disagree and 7 = strongly agree). Here are the 10 usefulness items:
- Using this product improves the quality of the work I do.
- Using this product gives me greater control over my work.
- This product enables me to accomplish tasks more quickly.
- This product supports critical aspects
- This product increases my productivity
- This product improves my job performance
- This product allows me to accomplish more work than would otherwise be possible.
- This product enhances my effectiveness on the job
- This product makes it easier to do my job
- Overall, I find this product useful in my job.
Davis had current users of software (from the late 80's and early 90's) answer the 20 TAM items as well as their current usage of the systems being examined (text editor and electronic mail) to see how well he could predict their usage.
Usefulness is 1.5 times more important than Ease of Use
Davis found the usefulness items were able to predict around 36% of actual reported usage. The usability items had a more indirect effect on use by impacting attitudes towards usage. Davis found that answers to the usefulness scores were about 1.5 times more important than usability scores in predicting actual use.
In short, you can explain a lot of whether people will accept or reject a technology based on how useful and usable it is.
Measuring Ease of Use
As it happens, several of the 10 ease of use items Davis used are the same or very similar to those used in the System Usability Scale (SUS)
. So when measuring product usability it's better to just use the SUS which has been studied refined and is widely used. For website usability use the SUPR-Q
which has 4 items that effectively substitute for the SUS, are designed specifically for websites and are normalized against a database of 200 websites.
It would be great to ask people a few questions to see if they would both use and then purchase a product. Asking people what they want in a new product is notoriously unreliable--as Steve Jobs has famously noted.
We can and should be critical of data based on users' predictions of their future behavior from focus groups, surveys or even the most complex statistical analysis. However, being critical is different than dismissing data altogether. Behavioral metrics are fickle but they're often the only data we have and they are often quite valid predictors. I wonder what the usefulness scores of Netflix would have looked like prior to the recent maligned price and service changes
Here's how to use this information:
- Consider measuring usefulness: by presenting the 10 TAM items to your users or potential users of a product
- Measure over time: See how changes in features correspond to changes in reported use.
- Predict Usage and adoption: Ask the TAM prior to release of a product then after users have some time with it. See how well their usage was predicted by the TAM items.
- Generate Benchmarks: Collect enough Technology Acceptance data to create an internal set of benchmarks when prioritizing products as I've done with usability and Net promoter benchmarks.