Marcus Andrey Study
Ler em português Library

Framework · Research · 3 min read

Margin of error and sample size

One formula that tells you how far a survey percentage is likely to be from the truth, and, turned around, how many responses and invitations you need for the precision a decision asks for.

nResponses

Finished answers, not invitations. It sits under a square root.

e.g. 400 people
pObserved share

The result as a decimal. Plan with 0.5 before you have one.

e.g. 62% is 0.62
zConfidence

How often the range would hold the true value.

1.96 for 95%, the usual choice
±Margin

Half the confidence interval. Halving it takes four times the responses.

e.g. ±4.8 points: 57.2% to 66.8%

MOE = z × √( p × (1 − p) ÷ n ) · n = z² × p × (1 − p) ÷ MOE²

Three inputs give the margin of error, and the same formula turned around gives the responses you need.

Use it when

  • Someone asks how many responses a survey needs before it goes out.
  • You report a percentage from a survey, or a success rate from a usability study, and need to say how precise it is.
  • A report compares two groups or two options and calls a small gap a win.

How to use it

  1. Start from the decision. Ask: "If the true number were 10 points lower than what we saw, would we decide differently?" If not, ±10 is enough. Output: the margin you can live with, at 95% confidence.
  2. Turn the margin into completes. Use n = z² × p × (1 − p) ÷ MOE², with z = 1.96 for 95% and p = 0.5 before you have results, because 0.5 gives the widest margin. Always round up. Output: the number of finished answers you need.
  3. Turn completes into invitations. Send the survey to a slice of the list first (a soft launch), measure the response rate, and divide the completes by it. Output: the number of invitations to send.
  4. Report each result as a range. Use MOE = z × √(p × (1 − p) ÷ n) with the share you observed. For small samples, or shares close to 0% or 100%, use the adjusted-Wald interval instead. Output: "62% ± 4.8 points (95%, n = 400)".
  5. Give subgroups and gaps their own margin. A subgroup has a bigger margin than the whole sample, and the gap between two numbers has a margin about twice that of each. Output: a separate sample size for every group you plan to compare.
  6. Say what the margin doesn't cover. It assumes a random sample. Who couldn't be reached, who chose not to answer and leading questions are not in it. Output: one caveat sentence next to the number.

Worked example

A team plans a survey for a bill-splitting app and needs the share of people who split bills with flatmates. A 10-point error would change the decision, a 5-point error wouldn't. At 95% and p = 0.5, ±5 needs 385 completes; the soft launch shows a 10% response rate, so they send 3,850 invitations. 400 people answer and 62% say yes: ±4.8 points, so between 57.2% and 66.8%. Change the numbers below to see how the margin moves.

The lesson's two examples, live. Change any number: the left side checks a result you have, the right side plans a survey you're about to send.

A result you have
A survey you're planning

Watch out for

  • Precision with no margin. "73.4%" from 30 responses looks exact and isn't. Put the n and the margin next to every percentage.
  • The whole sample's margin on a subgroup. ±3 for 1,067 people becomes about ±8 for a subgroup of 160.
  • Calling a small gap a win. "Option A 52%, option B 47%, ±3" is not a clear win: the margin on the gap is about twice as big.
  • A margin for an opt-in sample. The formula assumes random sampling. A self-selected sample can be far off, and the margin won't show it.

Template

Copy this into the research plan before fieldwork, and the report line into the findings.

Decision this number feeds: ...
Margin we can live with: ± ... points at 95%
  (If the true number were that much lower, would we decide differently? No.)
Completes needed: n = 1.96² x p x (1 - p) / MOE²  (p = 0.5 if unknown; round up) = ...
Groups to compare: ...  (each needs its own n)
Response rate (soft launch): ...%
Invitations: completes / response rate = ...

Report line:
...% ± ... points (95%, n = ...), sampling error only.
Respondents were ..., which over- or under-represents ...
Source
Margin of error definition: Pew Research Center (Andrew Mercer, 2016). Adjusted-Wald interval: Alan Agresti and Brent Coull (1998). Response-rate soft launch: Jeff Sauro (MeasuringU, 2011)

"See on Amazon" links are affiliate links. As an Amazon Associate I earn from qualifying purchases, at no extra cost to you.

Saved only on this device.

↑ ↓ to move · Enter to open · Esc to close