Framework · Research · 3 min read
Margin of error and sample size
One formula that tells you how far a survey percentage is likely to be from the truth, and, turned around, how many responses and invitations you need for the precision a decision asks for.
Finished answers, not invitations. It sits under a square root.
e.g. 400 peopleThe result as a decimal. Plan with 0.5 before you have one.
e.g. 62% is 0.62How often the range would hold the true value.
1.96 for 95%, the usual choiceHalf the confidence interval. Halving it takes four times the responses.
e.g. ±4.8 points: 57.2% to 66.8%MOE = z × √( p × (1 − p) ÷ n ) · n = z² × p × (1 − p) ÷ MOE²
Use it when
- Someone asks how many responses a survey needs before it goes out.
- You report a percentage from a survey, or a success rate from a usability study, and need to say how precise it is.
- A report compares two groups or two options and calls a small gap a win.
How to use it
- Start from the decision. Ask: "If the true number were 10 points lower than what we saw, would we decide differently?" If not, ±10 is enough. Output: the margin you can live with, at 95% confidence.
- Turn the margin into completes. Use n = z² × p × (1 − p) ÷ MOE², with z = 1.96 for 95% and p = 0.5 before you have results, because 0.5 gives the widest margin. Always round up. Output: the number of finished answers you need.
- Turn completes into invitations. Send the survey to a slice of the list first (a soft launch), measure the response rate, and divide the completes by it. Output: the number of invitations to send.
- Report each result as a range. Use MOE = z × √(p × (1 − p) ÷ n) with the share you observed. For small samples, or shares close to 0% or 100%, use the adjusted-Wald interval instead. Output: "62% ± 4.8 points (95%, n = 400)".
- Give subgroups and gaps their own margin. A subgroup has a bigger margin than the whole sample, and the gap between two numbers has a margin about twice that of each. Output: a separate sample size for every group you plan to compare.
- Say what the margin doesn't cover. It assumes a random sample. Who couldn't be reached, who chose not to answer and leading questions are not in it. Output: one caveat sentence next to the number.
Worked example
A team plans a survey for a bill-splitting app and needs the share of people who split bills with flatmates. A 10-point error would change the decision, a 5-point error wouldn't. At 95% and p = 0.5, ±5 needs 385 completes; the soft launch shows a 10% response rate, so they send 3,850 invitations. 400 people answer and 62% say yes: ±4.8 points, so between 57.2% and 66.8%. Change the numbers below to see how the margin moves.
The lesson's two examples, live. Change any number: the left side checks a result you have, the right side plans a survey you're about to send.
Watch out for
- Precision with no margin. "73.4%" from 30 responses looks exact and isn't. Put the n and the margin next to every percentage.
- The whole sample's margin on a subgroup. ±3 for 1,067 people becomes about ±8 for a subgroup of 160.
- Calling a small gap a win. "Option A 52%, option B 47%, ±3" is not a clear win: the margin on the gap is about twice as big.
- A margin for an opt-in sample. The formula assumes random sampling. A self-selected sample can be far off, and the margin won't show it.
Template
Copy this into the research plan before fieldwork, and the report line into the findings.
Decision this number feeds: ...
Margin we can live with: ± ... points at 95%
(If the true number were that much lower, would we decide differently? No.)
Completes needed: n = 1.96² x p x (1 - p) / MOE² (p = 0.5 if unknown; round up) = ...
Groups to compare: ... (each needs its own n)
Response rate (soft launch): ...%
Invitations: completes / response rate = ...
Report line:
...% ± ... points (95%, n = ...), sampling error only.
Respondents were ..., which over- or under-represents ...
- Quantifying the User ExperienceJeff Sauro & James R. LewisBook · 2nd ed. 2016 (Amazon)
- Surveys That Work: A Practical Guide for Designing and Running Better SurveysCaroline JarrettBook · 2021 (Amazon)
- Understanding the margin of errorPew (external site)
- Confidence Intervals, Margins of Error, and Confidence Levels in UXNN/g (external site)
- Margins of Error in Usability TestsMeasuringU (external site)
- Free calculatorsMeasuringU (external site)
"See on Amazon" links are affiliate links. As an Amazon Associate I earn from qualifying purchases, at no extra cost to you.