Marcus Andrey Study
Ler em português Library

UX Research · Step 5 · 8 min read

Sample size and margin of error

How many responses a survey needs depends on how precise the decision needs the number to be. What a margin of error means, the one formula behind it, worked examples, why halving the error costs four times the responses, and what the margin of error doesn't cover.

"How many people do we need?" has no answer until you say how wrong you can afford to be. A survey of 100 people that says 60% is really saying "somewhere between about 50% and 70%". Sometimes that's plenty: if the decision only changes below 30%, you're done. Sometimes it's useless. The margin of error is how you turn "is this enough?" from a feeling into a sum.

What you'll be able to do after reading:

  • Explain a margin of error in one plain sentence
  • Calculate it for a survey result, by hand
  • Work backwards from the precision you need to the responses and invitations to plan for
  • Say what the margin of error doesn't protect you from

1. What a margin of error means

A survey asks a sample, not everyone, so its result will almost never match the true value exactly. The margin of error says how far off it is likely to be. Pew Research Center's definition is the clearest: a margin of ±3 percentage points at the 95% confidence level "means that if we fielded the same survey 100 times, we would expect the result to be within 3 percentage points of the true population value 95 of those times" (Pew, Understanding the margin of error, 2016).

So "62% ± 5" is a range, 57% to 67%, and a level of confidence that the range holds the true value. NN/g explains the same idea for UX metrics, where the range is called a confidence interval and the margin of error is half its width (NN/g, Confidence Intervals, Margins of Error, and Confidence Levels in UX).


2. The formula, for a percentage

For a yes-or-no answer, or any share of people, the usual formula is:

MOE = z × √( p × (1 − p) ÷ n )
  • n is the number of responses.
  • p is the share you observed, as a decimal: 62% is 0.62.
  • z sets the confidence level: 1.96 for 95%, the usual choice.

Worked example. 400 people answer, and 62% say they split bills with flatmates.

  1. p × (1 − p) = 0.62 × 0.38 = 0.2356
  2. ÷ n: 0.2356 ÷ 400 = 0.000589
  3. Square root: 0.0243
  4. × 1.96 = 0.048, so ±4.8 percentage points: between 57.2% and 66.8%.

Before you have results, plan with p = 0.5. It gives the widest margin, so it's the safe assumption: at n = 400, 1.96 × √(0.25 ÷ 400) = ±4.9. If the result lands far from 50%, the real margin will be smaller: at 20%, the same 400 responses give ±3.9.

One caution: this simple formula works poorly for small samples and for shares close to 0% or 100%. For those, use an adjusted method such as the adjusted-Wald interval; MeasuringU's free confidence interval calculator for a completion rate gives it, next to the simple one (MeasuringU, calculators).


3. The curve: why the next 100 responses matter less

Margin of error, in percentage points, by number of responses

  1. n = 10: ±31.0
  2. n = 25: ±19.6
  3. n = 50: ±13.9
  4. n = 100: ±9.8
  5. n = 200: ±6.9
  6. n = 400: ±4.9
  7. n = 1,000: ±3.1
  8. n = 2,000: ±2.2
Responses (n)95% confidence level, p = 0.5
The margin of error falls fast at first and slowly after: to halve it, you need four times the responses (100 → 400 takes it from ±9.8 to ±4.9).Values computed with MOE = 1.96 × √(p(1 − p)/n), for p = 0.5, the worst case, on a simple random sample. Chart: Marcus Andrey, CC BY 4.0.

Because n sits under a square root, precision gets expensive fast. To halve the margin of error, you need four times the responses: 100 responses give about ±10, 400 give about ±5, 1,600 give about ±2.5. Jeff Sauro states the same rule for usability metrics: it's an inverse square-root relationship (MeasuringU, Margins of Error in Usability Tests, 2009).

Two things follow:

  • The first responses buy the most. Going from 25 to 100 cuts the margin from about ±20 to ±10. Going from 1,000 to 2,000 cuts it from ±3.1 to ±2.2.
  • The population size barely matters, as long as the sample is a small part of it. The formula has no "population" in it. A random sample of 1,000 is about as precise for 50,000 users as for 50 million.

4. Plan backwards from the decision

  1. ±5Choose the margin

    The widest error the decision can live with.

    Here: 5 percentage points, at 95%
  2. 384Completes you need

    n = 1.96² × 0.5 × 0.5 ÷ 0.05²

    Round up: 385 finished answers
  3. 10%Your response rate

    Measured in a soft launch to a slice of the list.

    Not guessed, not hoped for
  4. 3,850Invitations to send

    385 ÷ 0.10

    Can't reach that many? Accept a wider margin
Plan backwards from the decision: margin first, then completes, then invitations.Diagram: Marcus Andrey, CC BY 4.0. Soft-launch method: Jeff Sauro, How To Estimate A Survey Response Rate (MeasuringU, 2011).

Turn the formula around to get the sample size you need:

n = z² × p × (1 − p) ÷ MOE²

With p = 0.5 and 95% confidence: ±10 needs 97, ±5 needs 385, ±3 needs 1,068 (always round up). Pew's own example matches: a simple random sample of 1,067 has a margin of about ±3 points (Pew, 2016). The Margin of error and sample size card has both formulas, a plan template and a calculator that does the sums for you.

Then plan for the people who won't answer. Sauro's method is to send the survey to a slice of the list first, a soft launch, measure the response rate, and use it to estimate the full send (MeasuringU, How To Estimate A Survey Response Rate, 2011). At a 10% response rate, 385 completes means about 3,850 invitations.

And choose the margin from the decision, not from habit. Ask: "If the true number were 10 points lower than what we saw, would we decide differently?" If not, ±10 is enough, and 100 responses will do.


5. Groups and comparisons need more

Two traps catch almost every first survey report.

  • Subgroups have their own, bigger margin. The ±3 of a 1,067-person poll applies to the whole sample. Pew's example: a subgroup of about 160 people has a margin of about ±8 points (Pew, 2016). If you plan to compare new and long-time users, each group needs its own sample size.
  • A difference between two numbers has a bigger margin than each number. Pew notes that the margin for the gap between two candidates in one poll is generally about twice the margin for each. So "Option A 52%, option B 47%, ±3" is not a clear win.

For differences between two versions of a design, use a test built for comparing, which is what the next lesson's A/B tests do.


6. What the margin of error doesn't cover

The formula assumes a random sample from the population you care about. It says nothing about the people who couldn't be reached, the ones who chose not to answer, or a leading question. Pew says it plainly: sampling error is only one of many possible sources of error, and the others are harder to measure and rarely reported (Pew, 2016). Pew also notes that weighting a sample to look like the population widens the real margin, through what's called the design effect.

So report it honestly: "±5 points at 95%, for sampling error only; respondents were people who opened the app in March, which over-represents active users." That one sentence links this lesson with Bias in surveys.

And remember what it's for. Qualitative studies, like Usability testing with five people, aren't trying to estimate a percentage, so a margin of error isn't the right test for them. For quantitative usability metrics, How many participants has the numbers.


7. Common mistakes

  • Reporting "73.4%" from 30 responses, with no margin.
  • Comparing two subgroups using the whole sample's margin.
  • Calling a 4-point gap a win when the margin on the gap is ±6.
  • Quoting a margin of error for an opt-in, self-selected sample as if it were random.
  • Planning the sample size and forgetting the response rate.
  • Waiting for 1,000 responses when the decision was clear at 100.

8. Vocabulary

Term Meaning
Sample The people who actually answered
Population Everyone you want the result to describe
Margin of error How far the sample's result is likely to be from the true value
Confidence level How often the range would contain the true value if you repeated the survey; usually 95%
Confidence interval The range: result minus and plus the margin
Response rate Completed answers divided by invitations sent
Soft launch Sending to a slice of the list first, to check the survey and measure response
Design effect The loss of precision caused by weighting or a complex sample

9. Try it (no tools needed, or a phone calculator)

  1. A survey of 100 people finds 30% use a feature weekly. Calculate the margin at 95%: 1.96 × √(0.3 × 0.7 ÷ 100).
  2. How many responses would you need to get the margin down to ±5, assuming p = 0.5?
  3. Your response rate is 8%. How many invitations is that?

(Answers: about ±9 points; 385; about 4,813.)

Reflection questions: What's the last number your team decided on? Would the decision have survived a margin of ±10?


10. If you already practise

  • Put the margin of error next to every percentage in reports, and the n next to every chart.
  • Set the precision target in the research plan, with the decision it serves, before fieldwork.
  • Plan subgroup sizes explicitly when the question is a comparison.
  • Use a soft launch on every large send: it measures the response rate and catches a broken question before it reaches everyone.

Checklist

Learn more

Books, courses and videos to go deeper

Book links go to Amazon. As an Amazon Associate I earn from qualifying purchases, at no extra cost to you.

Bias in surveys · Margin of error and sample size · How many participants · Measuring usability · Screeners and sampling · Analytics and A/B tests for designers

Saved only on this device.

↑ ↓ to move · Enter to open · Esc to close