UX Research · Step 4 · 8 min read
Measuring usability
Usability can be measured, if you're clear about what you measure. The ISO definition in three parts, the metrics for each (success rate, time on task, the Single Ease Question and the System Usability Scale), how to score SUS by hand, what 68 means, and why numbers need many more people than findings do.
"Users found it easy" is an opinion. "8 of 10 finished, in a median of 74 seconds, and rated it 77.5 on SUS" is a measurement. Numbers don't replace watching people: they tell you how much, and only watching tells you why. But when you need to compare two versions, track a product across releases or convince someone who wasn't in the room, a small set of standard measures does what a quote can't.
What you'll be able to do after reading:
- Explain the ISO 9241-11 definition of usability in its three parts
- Choose a metric for effectiveness, efficiency and satisfaction
- Run the SEQ after a task and score a SUS questionnaire by hand
- Interpret a SUS score, and know how many participants a number needs
1. What "usable" means, officially
The international standard for usability is ISO 9241-11 ("Ergonomics of human-system interaction, Part 11: Usability: Definitions and concepts"), revised in 2018 (ISO 9241-11:2018). John Brooke, who chaired the group that wrote its first version, summarises its point: there's no property of a product you can call usability on its own. Something is usable for specified users, doing specified tasks, in a specified context, and the standard breaks it into three components (Brooke, SUS: A Retrospective, Journal of Usability Studies, 2013):
- Effectiveness: whether people can complete their tasks and achieve their goals.
- Efficiency: how much effort and resource they spend to get there.
- Satisfaction: how comfortable they feel while doing it.
- EffectivenessDid they get there?
Whether people complete the task and reach the goal, correctly.
Success rate: 7 of 10 added the bill with the right amounts.
- EfficiencyWhat did it cost them?
The time, steps and effort spent to reach the goal.
Time on task, wrong paths, errors: median 74 seconds.
- SatisfactionHow did it feel?
How comfortable people were along the way, in their own rating.
SEQ after each task; SUS at the end of the session.
All three matter. Brooke's own example: a system people complete tasks with, but only through great effort and with everyone unhappy, isn't usable. Neither is one that's enjoyable but where nothing gets done.
2. Effectiveness and efficiency
Success rate is the simplest usability metric: the percentage of participants who complete a task (NN/g, Nielsen and Budiu, Success Rate, 2001). Define success before the sessions, per task (see Tasks and scenarios), and score everyone the same way.
Time on task is the main efficiency measure, along with errors and wrong paths. Two cautions. Report the median, because a few very slow sessions pull the average up. And don't measure time in a think-aloud session: talking slows people down, which NN/g notes contaminates task times (Nielsen and Budiu, 2001). Measuring is its own study, with its own sessions.
3. Satisfaction after each task: the SEQ
The Single Ease Question asks one thing, straight after each task: "Overall, how difficult or easy was the task to complete?", on a 7-point scale, usually with only the two ends labelled. Asking at once means the task is fresh and later tasks don't colour the answer (NN/g, Laubheimer, Beyond the NPS, 2018). Across more than 400 tasks and 10,000 users, Jeff Sauro found the average SEQ hovers around 5.5 (MeasuringU, Sauro, 10 Things to Know About the SEQ, 2012, average updated 2019). A task well below that is a good place to look first.
4. Satisfaction for the whole thing: SUS
The System Usability Scale was created by John Brooke at DEC in the mid-1980s as a "quick and dirty" way to measure how people perceive a system's usability, and it was made freely available (Brooke, 2013). It's ten statements, each answered on five points from strongly disagree (1) to strongly agree (5), given once at the end of a session. The odd items are positive ("I thought the system was easy to use"); the even items are negative ("I found the system unnecessarily complex").
Scoring (Sauro, Measuring Usability with the SUS, MeasuringU, 2011):
- Odd items: subtract 1 from the answer.
- Even items: subtract the answer from 5.
- Each item now scores 0 to 4. Add them up and multiply by 2.5, for a score from 0 to 100.
| # | Statement | Answer (1–5) | Rule | Points |
|---|---|---|---|---|
| 1 | I think that I would like to use this system frequently. | 4 | answer − 1 | 3 |
| 2 | I found the system unnecessarily complex. | 2 | 5 − answer | 3 |
| 3 | I thought the system was easy to use. | 4 | answer − 1 | 3 |
| 4 | I think that I would need the support of a technical person to be able to use this system. | 1 | 5 − answer | 4 |
| 5 | I found the various functions in this system were well integrated. | 4 | answer − 1 | 3 |
| 6 | I thought there was too much inconsistency in this system. | 2 | 5 − answer | 3 |
| 7 | I would imagine that most people would learn to use this system very quickly. | 5 | answer − 1 | 4 |
| 8 | I found the system very cumbersome to use. | 2 | 5 − answer | 3 |
| 9 | I felt very confident using the system. | 3 | answer − 1 | 2 |
| 10 | I needed to learn a lot of things before I could get going with this system. | 2 | 5 − answer | 3 |
| Sum of points | 31 | |||
| × 2.5 = SUS score | 77.5 |
The ×2.5 isn't science. Brooke explains it was a marketing decision at DEC: managers and engineers understood a 0 to 100 scale more easily than 10 to 50 (Brooke, 2013). The downside is that people read SUS scores as percentages, and they aren't.
Interpreting a score. Sauro's data from 500 studies gives an average SUS of 68; above it is above average, below it below (MeasuringU, 2011). Percentiles are the honest way to report it: a 74 is better than about 70% of products, and anything below 51 sits in the bottom 15%. NN/g adds that you need 80 or more to be in the top 10% (Laubheimer, 2018). Sauro also notes that items 4 and 10 form a learnability sub-scale, and the other eight a usability one. The SUS card has the ten items to copy and a calculator that scores one person's answers.
5. How many people a number needs
A SUS of 77.5 from three people tells you very little: one more participant could move it by ten points. Every metric is an estimate of what your whole user population would score, and its precision depends on the sample. NN/g's guideline for most quantitative usability studies is around 40 participants, and every metric should be reported with its confidence interval (NN/g, Budiu and Moran, 2021). The reasoning is in How many participants, and the maths in Sample size and margin of error.
That doesn't make metrics useless in small tests. Asking the SEQ in a five-person qualitative test is fine, as a conversation starter ("You gave that a 3. Tell me about it"), not as a result. The rule is simple: findings from five, numbers from forty.
Measures earn their keep over time. Run the same tasks and SUS on each major release, with similar participants, and you get a benchmark: a line that shows whether the product is getting easier to use. That links straight to Defining success and to the HEART-style goals your team already tracks.
6. Common mistakes
- Reporting a SUS score as a percentage ("70% usable").
- Rewording or dropping SUS items, then comparing with the 68 benchmark.
- Averaging task times instead of reporting the median.
- Measuring time in think-aloud sessions.
- Percentages and averages from five people, with no confidence interval.
- Asking the SEQ after the follow-up questions, when your questions have already shaped the answer.
7. Vocabulary
| Term | Meaning |
|---|---|
| ISO 9241-11 | The international standard that defines usability: effectiveness, efficiency and satisfaction in a context of use |
| Success rate | The percentage of participants who complete a task |
| Time on task | How long a task takes; reported as a median |
| SEQ | Single Ease Question: one 7-point rating of difficulty, after each task |
| SUS | System Usability Scale: ten statements, scored 0 to 100, after the session |
| Percentile | Where a score sits compared with others: the 70th percentile beats 70% of them |
| Benchmark | The same measures repeated over time, to track change |
8. Try it (no tools needed)
- Answer the ten SUS statements yourself for an app you use every day. (The items are on MeasuringU's SUS page.)
- Score it by hand with the three rules. Then do it for an app you dislike.
- Where does each one sit against 68 and 80?
Reflection questions: Did the number match your feeling about each app? What would it take to trust a score like that for your own product?
9. If you already practise
- Pick three top tasks and run them with SUS on every major release: a simple benchmark is worth more than a fancy one-off.
- Use the original SUS wording. If you need another language, Brooke notes that early translations had no validation; look for a validated version, or check yours before comparing it with the English benchmarks.
- Report every metric with its sample size and confidence interval, and pair it with one finding that explains it.
- Sauro and Lewis's Quantifying the User Experience is the reference for the statistics behind all of this.
Checklist
Learn more
- ISO: ISO 9241-11:2018, Ergonomics of human-system interaction, Part 11: Usability: Definitions and concepts
- John Brooke, SUS: A Retrospective (Journal of Usability Studies 8(2), 2013) · SUS: A "quick and dirty" usability scale (in Usability Evaluation in Industry, 1996)
- MeasuringU: Measuring Usability with the System Usability Scale (SUS) (Sauro, 2011) · 10 Things to Know About the Single Ease Question (Sauro, 2012)
- NN/g: Success Rate: The Simplest Usability Metric (Nielsen and Budiu, 2001) · Beyond the NPS: SUS, NASA-TLX, and the Single Ease Question (Laubheimer, 2018) · How Many Participants for Quantitative Usability Studies (Budiu and Moran, 2021)
Books, courses and videos to go deeper
- Quantifying the User ExperienceJeff Sauro and James R. LewisBook · 2nd edition (Morgan Kaufmann, 2016) (Amazon)
- Measuring Usability with the System Usability ScaleMeasuringUfree, article (external site)
Book links go to Amazon. As an Amazon Associate I earn from qualifying purchases, at no extra cost to you.
Related
Usability testing with five people · Defining success · How many participants · Sample size and margin of error · Analytics and A/B tests for designers · Writing survey questions