Marcus Andrey Study
Ler em português Library

UX Research · Step 4 · 7 min read

How many participants

"Test with five users" is the best-known number in UX, and the most misread. Where it comes from, what the formula really says, what Faulkner found when she tested 60 people, how the answer changes for problems that are rarer, for different user groups and for measuring, and how to choose a number you can defend.

Five is a good answer to one specific question, and a bad answer to all the others. It's enough to find most of the problems that many of your users will hit, in one round of a qualitative test, with similar people. It isn't enough to find rare problems, to cover very different groups, or to put a number on anything. Knowing which question you're asking is what turns "five users" from a slogan into a decision.

What you'll be able to do after reading:

  • Explain the Nielsen and Landauer formula and the curve behind "five users"
  • Read the curve with its real assumptions, and see what changes when they don't hold
  • Use Faulkner's results to explain the risk of small samples
  • Choose a number of participants for qualitative tests, several user groups and quantitative studies

1. Where five comes from

In 1993, Jakob Nielsen and Tom Landauer published a mathematical model of how usability problems are found (Nielsen and Landauer, A mathematical model of the finding of usability problems, INTERCHI '93). In Nielsen's 2000 summary, the share of problems found with n participants is:

found(n) = 1 − (1 − L)ⁿ

where L is the proportion of problems one participant reveals. Across the projects they studied, the typical L was 31% (NN/g, Nielsen, Why You Only Need to Test with 5 Users, 2000).

One user finds 31% of problems (Nielsen's average)One user finds 10% (a harder-to-find set)

Line chart of the share of usability problems found, by number of participants, from the formula 1 minus (1 minus L) to the power n. With L = 0.31: 31% with 1 participant, 52% with 2, 67% with 3, 77% with 4, 84% with 5, 98% with 10 and over 99% with 15. With L = 0.10: 41% with 5, 65% with 10, 79% with 15.0%25%50%75%100%01351015Participants5 people: 84%10: 98%5 people: 41%
The share of problems found, by number of participants: 1 − (1 − L)n, drawn with the real values. With Nielsen and Landauer's average L of 31%, five people find 84% (Nielsen rounds it to 85%). If each person only meets 10% of the problems, five people find 41%. The curve is only as good as its L.Chart: Marcus Andrey, CC BY 4.0, from the formula in Nielsen and Landauer, A mathematical model of the finding of usability problems (INTERCHI 1993), as used in Nielsen, 2000.

The curve explains the advice. The first person teaches you almost a third of what there is to learn. The second repeats some of it and adds less. By the fifth, you're mostly watching the same problems again. Nielsen notes you'd need at least 15 people to find nearly all the problems, but recommends spending that budget on three rounds of five, fixing between rounds, because the goal is to improve the design, not to document its flaws.


2. The assumptions inside the curve

Raluca Budiu at NN/g spells out why five is fine for a qualitative study: its goal is to find problems, and any problem you see is real, the way one person falling in a pothole is enough to know it needs fixing. A qualitative test doesn't try to predict how many users will hit a problem (NN/g, Budiu, Why 5 Participants Are Okay in a Qualitative Study, but Not in a Quantitative One, 2021).

She also points at the weak spot: 31% is an average from projects in the early 1990s. If your design has been improved over many rounds, problems are harder to hit, L is lower, and you need more people. The formula still works; you just put in a different L.

Jeff Sauro at MeasuringU reframes the same maths in a way that's easier to use. Five users give you about an 85% chance of seeing a problem that affects 31% of users. A problem that affects 10% of users needs 18 participants for the same chance. And in software that's already released, he suggests the likelihood of a problem is probably closer to 10% or less (Sauro, Why you only need to test with five users (explained), MeasuringU, 2010).

Problems that affect… Participants for an 85% chance of seeing them
31% of users 5
20% of users 9
10% of users 18

3. What happens with real people

Laura Faulkner tested the five-user assumption directly. She ran 60 participants, then drew random groups from them to see what each group would have found. Some groups of five found 99% of the problems; others found only 55%. With ten people, the worst group still found 80%; with twenty, 95% (Faulkner, Beyond the five-user assumption, Behavior Research Methods, Instruments, & Computers, 2003).

  1. 5 people, luckiest set99%
  2. 5 people, unluckiest set55%
  3. 10 people, unluckiest set80%
  4. 20 people, unluckiest set95%

Share of all the problems found by 60 participants, in random groups drawn from them.

What five people find depends on which five. Laura Faulkner tested 60 people and drew random groups from them: some groups of five found 99% of the problems, others only 55%. More people mostly raise the floor: the worst group of ten found 80%, the worst of twenty, 95%.Chart: Marcus Andrey, CC BY 4.0, with the numbers from Faulkner, Beyond the five-user assumption, Behavior Research Methods, Instruments, & Computers 35 (2003).

So "five people find 85%" is an average. Your five might be a lucky set or an unlucky one, and you won't know which. That's not a reason to abandon small tests; it's a reason to run several of them, and to be modest about what one round proves.


4. Different groups, interviews, and numbers

Several user groups. The formula assumes people who are alike and use the product in similar ways. If you have very different groups (Nielsen's example: children and parents), test each one. Because the groups still share a lot, you need fewer per group: 3 to 4 each for two groups, and 3 each for three or more, never fewer than 3 per group (Nielsen, 2000).

Interviews. Interviews aim to understand experiences, which vary more than interface problems, so there's no formula. NN/g's advice is to start small, analyse as you go, and stop at saturation, when new interviews stop changing your themes (NN/g, Rosala, How Many Participants for a UX Interview?, 2021). See Short user interviews.

Measuring. Numbers need many more people. NN/g's guideline for most quantitative usability studies is 40 participants: that's what it takes to estimate a success rate within about 15 percentage points with 95% confidence (NN/g, Budiu and Moran, How Many Participants for Quantitative Usability Studies, 2021). If 70% succeed in that study, the real rate could be anywhere from 55% to 85%. With five people, the range is so wide that the number means almost nothing. That's the subject of Measuring usability and Sample size and margin of error.


5. Choosing your number

You want to… Start with
Find the main problems in a design, iteratively 5 per round, several rounds
Test two or three distinct user groups 3 to 4 per group (at least 3)
Catch rarer problems in a mature product More per round (9 to 18), or more rounds
Understand experiences in interviews 5 to 6, then continue until saturation
Measure success rate, time or satisfaction Around 40

And always recruit one or two extra: someone always cancels.


6. Common mistakes

  • Quoting "85%" as if every round of five finds it.
  • Using five for a mature product where problems are rare, and concluding it's fine.
  • Mixing very different users in one group of five.
  • Reporting percentages from five people ("80% succeeded").
  • One round of five, and no second round after the fixes.
  • Using the usability formula to size interview studies.

7. Vocabulary

Term Meaning
L The proportion of problems one participant reveals (31% in Nielsen and Landauer's data)
Problem discovery The goal of a qualitative test: finding what goes wrong
Iterative testing Several small rounds, with fixes in between
Saturation The point where new sessions stop adding new themes
Confidence interval The range where the real value probably lies, given your sample
Margin of error Half the width of the confidence interval

8. Try it (no tools needed)

  1. With a calculator, work out 1 − (1 − L)ⁿ for L = 0.31 and n = 1, 3, 5. Then for L = 0.10 and the same n.
  2. Compare with the chart: did you get 31%, 67%, 84%, and 10%, 27%, 41%?
  3. Think of your product: is it a new design or a mature one? Which L feels closer?

Reflection questions: If someone on your team says "we tested with five, so it's fine", what would you ask them? What would you need to know before agreeing?


9. If you already practise

  • Plan studies as rounds, not as a number: "three rounds of five over six weeks" is a better plan than "fifteen users".
  • After a round, count how many new problems the last two participants added. If it's still many, add a session.
  • When stakeholders push back on small samples, show them the Faulkner chart and the plan for the next round, not the 85%.
  • Sauro and Lewis's Quantifying the User Experience covers sample sizes for every kind of study, with the formulas.

Checklist

Learn more

Books, courses and videos to go deeper

Book links go to Amazon. As an Amazon Associate I earn from qualifying purchases, at no extra cost to you.

Usability testing with five people · Planning a usability test · Measuring usability · Sample size and margin of error · Screeners and sampling · Short user interviews

Saved only on this device.

↑ ↓ to move · Enter to open · Esc to close