Marcus Andrey Study
Ler em português Library

UX Research · Step 4 · 8 min read

Measuring usability

Usability can be measured, if you're clear about what you measure. The ISO definition in three parts, the metrics for each (success rate, time on task, the Single Ease Question and the System Usability Scale), how to score SUS by hand, what 68 means, and why numbers need many more people than findings do.

"Users found it easy" is an opinion. "8 of 10 finished, in a median of 74 seconds, and rated it 77.5 on SUS" is a measurement. Numbers don't replace watching people: they tell you how much, and only watching tells you why. But when you need to compare two versions, track a product across releases or convince someone who wasn't in the room, a small set of standard measures does what a quote can't.

What you'll be able to do after reading:

  • Explain the ISO 9241-11 definition of usability in its three parts
  • Choose a metric for effectiveness, efficiency and satisfaction
  • Run the SEQ after a task and score a SUS questionnaire by hand
  • Interpret a SUS score, and know how many participants a number needs

1. What "usable" means, officially

The international standard for usability is ISO 9241-11 ("Ergonomics of human-system interaction, Part 11: Usability: Definitions and concepts"), revised in 2018 (ISO 9241-11:2018). John Brooke, who chaired the group that wrote its first version, summarises its point: there's no property of a product you can call usability on its own. Something is usable for specified users, doing specified tasks, in a specified context, and the standard breaks it into three components (Brooke, SUS: A Retrospective, Journal of Usability Studies, 2013):

  • Effectiveness: whether people can complete their tasks and achieve their goals.
  • Efficiency: how much effort and resource they spend to get there.
  • Satisfaction: how comfortable they feel while doing it.
  • EffectivenessDid they get there?

    Whether people complete the task and reach the goal, correctly.

    Success rate: 7 of 10 added the bill with the right amounts.
  • EfficiencyWhat did it cost them?

    The time, steps and effort spent to reach the goal.

    Time on task, wrong paths, errors: median 74 seconds.
  • SatisfactionHow did it feel?

    How comfortable people were along the way, in their own rating.

    SEQ after each task; SUS at the end of the session.
Usability, as ISO 9241-11 breaks it down: three parts, always for specific users, goals and context. Each one has its own measure. A design that's quick and pleasant but where people don't finish isn't usable; neither is one they finish, slowly, hating it.Diagram: Marcus Andrey, CC BY 4.0, after ISO 9241-11:2018 as described by Brooke (2013). The numbers are an example.

All three matter. Brooke's own example: a system people complete tasks with, but only through great effort and with everyone unhappy, isn't usable. Neither is one that's enjoyable but where nothing gets done.


2. Effectiveness and efficiency

Success rate is the simplest usability metric: the percentage of participants who complete a task (NN/g, Nielsen and Budiu, Success Rate, 2001). Define success before the sessions, per task (see Tasks and scenarios), and score everyone the same way.

Time on task is the main efficiency measure, along with errors and wrong paths. Two cautions. Report the median, because a few very slow sessions pull the average up. And don't measure time in a think-aloud session: talking slows people down, which NN/g notes contaminates task times (Nielsen and Budiu, 2001). Measuring is its own study, with its own sessions.


3. Satisfaction after each task: the SEQ

The Single Ease Question asks one thing, straight after each task: "Overall, how difficult or easy was the task to complete?", on a 7-point scale, usually with only the two ends labelled. Asking at once means the task is fresh and later tasks don't colour the answer (NN/g, Laubheimer, Beyond the NPS, 2018). Across more than 400 tasks and 10,000 users, Jeff Sauro found the average SEQ hovers around 5.5 (MeasuringU, Sauro, 10 Things to Know About the SEQ, 2012, average updated 2019). A task well below that is a good place to look first.


4. Satisfaction for the whole thing: SUS

The System Usability Scale was created by John Brooke at DEC in the mid-1980s as a "quick and dirty" way to measure how people perceive a system's usability, and it was made freely available (Brooke, 2013). It's ten statements, each answered on five points from strongly disagree (1) to strongly agree (5), given once at the end of a session. The odd items are positive ("I thought the system was easy to use"); the even items are negative ("I found the system unnecessarily complex").

Scoring (Sauro, Measuring Usability with the SUS, MeasuringU, 2011):

  1. Odd items: subtract 1 from the answer.
  2. Even items: subtract the answer from 5.
  3. Each item now scores 0 to 4. Add them up and multiply by 2.5, for a score from 0 to 100.
#StatementAnswer (1–5)RulePoints
1I think that I would like to use this system frequently.4answer − 13
2I found the system unnecessarily complex.25 − answer3
3I thought the system was easy to use.4answer − 13
4I think that I would need the support of a technical person to be able to use this system.15 − answer4
5I found the various functions in this system were well integrated.4answer − 13
6I thought there was too much inconsistency in this system.25 − answer3
7I would imagine that most people would learn to use this system very quickly.5answer − 14
8I found the system very cumbersome to use.25 − answer3
9I felt very confident using the system.3answer − 12
10I needed to learn a lot of things before I could get going with this system.25 − answer3
Sum of points31
× 2.5 = SUS score77.5
A SUS sheet, filled in and scored. Odd items are positive, so their points are the answer minus 1; even items are negative, so their points are 5 minus the answer. Add them (31), multiply by 2.5, and this person's score is 77.5: above the average of 68. A study's score is the mean of everyone's.Example answers: Marcus Andrey, CC BY 4.0. Items and scoring: John Brooke, SUS (created at DEC in the mid-1980s, published 1996); scoring as in Brooke, SUS: A Retrospective (2013). Benchmarks: Sauro, MeasuringU (2011).

The ×2.5 isn't science. Brooke explains it was a marketing decision at DEC: managers and engineers understood a 0 to 100 scale more easily than 10 to 50 (Brooke, 2013). The downside is that people read SUS scores as percentages, and they aren't.

Interpreting a score. Sauro's data from 500 studies gives an average SUS of 68; above it is above average, below it below (MeasuringU, 2011). Percentiles are the honest way to report it: a 74 is better than about 70% of products, and anything below 51 sits in the bottom 15%. NN/g adds that you need 80 or more to be in the top 10% (Laubheimer, 2018). Sauro also notes that items 4 and 10 form a learnability sub-scale, and the other eight a usability one. The SUS card has the ten items to copy and a calculator that scores one person's answers.


5. How many people a number needs

A SUS of 77.5 from three people tells you very little: one more participant could move it by ten points. Every metric is an estimate of what your whole user population would score, and its precision depends on the sample. NN/g's guideline for most quantitative usability studies is around 40 participants, and every metric should be reported with its confidence interval (NN/g, Budiu and Moran, 2021). The reasoning is in How many participants, and the maths in Sample size and margin of error.

That doesn't make metrics useless in small tests. Asking the SEQ in a five-person qualitative test is fine, as a conversation starter ("You gave that a 3. Tell me about it"), not as a result. The rule is simple: findings from five, numbers from forty.

Measures earn their keep over time. Run the same tasks and SUS on each major release, with similar participants, and you get a benchmark: a line that shows whether the product is getting easier to use. That links straight to Defining success and to the HEART-style goals your team already tracks.


6. Common mistakes

  • Reporting a SUS score as a percentage ("70% usable").
  • Rewording or dropping SUS items, then comparing with the 68 benchmark.
  • Averaging task times instead of reporting the median.
  • Measuring time in think-aloud sessions.
  • Percentages and averages from five people, with no confidence interval.
  • Asking the SEQ after the follow-up questions, when your questions have already shaped the answer.

7. Vocabulary

Term Meaning
ISO 9241-11 The international standard that defines usability: effectiveness, efficiency and satisfaction in a context of use
Success rate The percentage of participants who complete a task
Time on task How long a task takes; reported as a median
SEQ Single Ease Question: one 7-point rating of difficulty, after each task
SUS System Usability Scale: ten statements, scored 0 to 100, after the session
Percentile Where a score sits compared with others: the 70th percentile beats 70% of them
Benchmark The same measures repeated over time, to track change

8. Try it (no tools needed)

  1. Answer the ten SUS statements yourself for an app you use every day. (The items are on MeasuringU's SUS page.)
  2. Score it by hand with the three rules. Then do it for an app you dislike.
  3. Where does each one sit against 68 and 80?

Reflection questions: Did the number match your feeling about each app? What would it take to trust a score like that for your own product?


9. If you already practise

  • Pick three top tasks and run them with SUS on every major release: a simple benchmark is worth more than a fancy one-off.
  • Use the original SUS wording. If you need another language, Brooke notes that early translations had no validation; look for a validated version, or check yours before comparing it with the English benchmarks.
  • Report every metric with its sample size and confidence interval, and pair it with one finding that explains it.
  • Sauro and Lewis's Quantifying the User Experience is the reference for the statistics behind all of this.

Checklist

Learn more

Books, courses and videos to go deeper

Book links go to Amazon. As an Amazon Associate I earn from qualifying purchases, at no extra cost to you.

Usability testing with five people · Defining success · How many participants · Sample size and margin of error · Analytics and A/B tests for designers · Writing survey questions

Saved only on this device.

↑ ↓ to move · Enter to open · Esc to close