UX Research · Step 5 · 8 min read
Bias in surveys
A survey can be wrong in two ways: it can ask the wrong people, or it can ask the right people in a way that bends their answers. The main biases by name, real experiments that show how big they are, and what you can do about each one.
A biased survey doesn't look biased. It looks like a clean chart. That's what makes bias dangerous: a percentage with a decimal point feels like a fact, whoever answered and however the question was put. You can't remove every bias. You can know their names, design against the big ones, and say honestly which ones are left.
What you'll be able to do after reading:
- Tell errors about who answers from errors about how they answer
- Name ten common survey biases and what each one does
- Show with real numbers how much wording and order can move results
- Reduce each bias in the design, and report the ones you couldn't
1. Two ways to be wrong
Survey methodologists group everything that can go wrong under one name, Total Survey Error: the idea is to consider all the errors together and make choices that keep the total low (Caroline Jarrett, Effortmark). Jarrett's Survey Octopus maps those errors onto the steps of a survey; for a designer, two big families are enough to start:
- Who answers. The people who could be asked, were asked and chose to answer may not be the people you want to know about. Pew gives a plain example: people without internet access can't take online surveys (Pew, Understanding the margin of error, 2016).
- How they answer. The right people, asked in a way that bends their answer: the wording, the order, the scale, who's asking.
More responses fix neither. Ten thousand answers from the wrong people, or to a leading question, are ten thousand wrong answers, with a smaller margin of error. As Jeff Sauro puts it, it's much better to ask a few of the right people than a lot of the wrong ones (MeasuringU, 2010).
2. The catalogue
| Bias | What happens | In the bill-splitting survey | How to reduce it |
|---|---|---|---|
| Coverage | The people you can reach aren't the people you want to know about | An in-app survey never reaches people who gave up on the app | Name the group first; recruit former users too, by e-mail |
| Non-response | The people who answer differ from the ones who don't | Fans answer; people with a bad settle-up ignore it | Keep it short, send one reminder, compare who answered with who you invited |
| Acquiescence | A tendency to agree with any statement | "Reminders are helpful. Agree or disagree?" | Offer a choice between two statements instead of agree or disagree |
| Social desirability | Answers that look good instead of answers that are true | Few admit they never paid a friend back | Anonymous and self-completed; word it so the honest answer is normal |
| Question order | Earlier questions change how later ones are read | Asking about a fight over money, then about overall satisfaction | General before specific; keep the same order when you track over time |
| Option order | On screen, people pick options near the top more often | "Message" always first in the list of reminder methods | Shuffle lists with no natural order; keep scales in order |
| Wording | Small word changes move the answers | "Nagging reminders" against "reminders" | Plain, neutral words; test two versions on two halves of the sample |
| Sponsor | Knowing who asks colours the answers | The app's logo and "help us improve" in the invitation | A neutral invitation; say who you are, without selling |
| Memory | People misremember when and how often things happened | "How many times this year did you settle up?" | A short, recent window: "in the past 7 days" |
| Satisficing | Tired respondents give the first acceptable answer | The same column ticked down a 15-row grid | Fewer questions, fewer grids, and don't make everything required |
You don't need to memorise the table; you need to recognise the pattern when you see it. A few deserve a closer look.
Acquiescence. Pew reports that less educated and less informed respondents are more likely to agree with agree-disagree statements, and more so when an interviewer is present. Their fix is to offer a choice between alternative statements instead (Pew Research Center).
Social desirability. People understate drinking, drug use and tax evasion, and overstate church attendance, charity and voting. Pew words its voting question to make "no" easy: "did things come up that kept you from voting, or did you happen to vote?" Self-completed web and paper surveys show less of this bias than phone or face-to-face ones. NN/g adds that asking for names and other identifiers increases it (NN/g, Rating Scales in UX Research, 2020).
Satisficing. When answering takes more effort than people want to give, they settle for an answer that seems good enough instead of the best one. Jon Krosnick described this response strategy in 1991, and it's why long surveys and big grids produce the same column ticked all the way down (Krosnick, Applied Cognitive Psychology, 1991).
Sponsor and memory. Knowing who sent the survey colours the answers, especially about brands. And people misplace events in time, remembering them as earlier or more recent than they were, which Sauro calls telescoping (MeasuringU, 9 Biases That Affect Survey Responses, 2016).
3. How big these effects are
These aren't small effects at the edge of statistics. Pew tests them by writing two versions of a question and giving each to a random half of the sample: because the halves are alike, any difference comes from the wording (Pew Research Center). Some of the results Pew reports:
- Wording. In 2003, 68% favoured military action in Iraq "to end Saddam Hussein's rule". Adding "even if it meant that U.S. forces might suffer thousands of casualties" dropped it to 43%.
- Emotional words. In 2005, 51% favoured letting doctors "give terminally ill patients the means to end their lives"; 44% favoured letting them "assist terminally ill patients in committing suicide". Same policy, different words.
- Order. In 2008, 88% said they were dissatisfied with the way things were going in the country when the question came right after one about the president's approval; 78% without it.
- Open or closed. 58% against 35% for "the economy", depending on whether it was offered in a list (see Writing survey questions).
Your product survey won't be about war. But "nagging reminders" against "reminders", or a satisfaction question right after a question about a failed payment, work the same way.
4. Order: what comes before changes what comes after
Pew describes two kinds of order effect. In a contrast effect, answers move apart because of the order; in an assimilation effect, they move closer together. A specific question asked before a general one, like happiness with your marriage before happiness overall, tends to produce contrast: people leave the specific part out of the general answer (Pew Research Center).
Options have an order too. Pew notes that in self-completed surveys people tend to pick options near the top of a list (a primacy effect), and on the phone, options heard last (recency). What to do:
- General before specific. Ask overall satisfaction before the questions about one bad experience.
- Shuffle lists that have no natural order, so every option gets the top spot equally often. Pew notes this doesn't remove the effect; it spreads it evenly.
- Keep scales in order, but you can reverse a scale for half the respondents, as Pew does on some questions.
- Keep the order fixed when you compare over time, or the change you see may be a change in context.
- Put sensitive and demographic questions near the end, unless you need them to screen people. Sauro also notes that questions which bring back bad memories can sour the answers that follow (MeasuringU, 2016).
5. Who answered?
The two "who" biases are the ones teams forget, because the data looks fine.
- Coverage. An in-app survey reaches only people who still open the app. If your question is why people leave, the people who left can't answer it.
- Non-response. Of the people you invite, the ones who answer are different: often more engaged, happier, or angrier. NN/g quotes researchers who call surveys the hardest method to do well and the easiest to launch, and advises starting one only with enough time to do it with rigour (NN/g, Should You Run a Survey?, 2024).
Three habits help:
- Write down the population before you write a question: "people who added at least one shared expense in the last 90 days".
- Compare respondents with invitees on what you already know, such as plan, country, months since sign-up. If the respondents are mostly long-time users, say so.
- Recruit the missing group on purpose, for example former users by e-mail, instead of hoping they'll turn up in the in-app survey.
6. Common mistakes
- Believing a big sample cancels bias. It shrinks the margin of error, not the bias.
- Agree-disagree statements for everything.
- A satisfaction question placed right after the questions about problems.
- Asking for name and e-mail at the start of a survey about sensitive behaviour.
- Reporting "78% of users" when it's 78% of the people who chose to answer.
- Changing one word in a tracked question, then celebrating the trend.
7. Vocabulary
| Term | Meaning |
|---|---|
| Total Survey Error | All the errors a survey can have, considered together |
| Coverage error | The people you can reach aren't the population you care about |
| Non-response bias | The people who answer differ from those who don't |
| Acquiescence | A tendency to agree with statements |
| Social desirability | Giving the answer that looks good instead of the true one |
| Satisficing | Giving a good-enough answer to save effort |
| Order effect | Earlier questions changing how later ones are answered |
| Primacy, recency | Picking options at the start, or at the end, of a list |
| Split-ballot test | Two versions of a question, each given to a random half |
8. Try it (no tools needed)
- Take a survey you've received recently.
- Who could not have answered it? Write down one group.
- Find one question whose answer might change if the question before it were different.
Reflection questions: If the survey's results were published tomorrow, which bias would you mention in the footnote? Would anyone in your team read the footnote?
9. If you already practise
- Add a "who answered" table to every survey report: invited, responded, and how they compare on what you know.
- When a question matters, run a split-ballot test of two wordings in the pilot, the way Pew does at scale.
- Randomise unordered lists by default in your survey tool, and turn it off on purpose, not by accident.
- Keep a limitations section in the report. Naming the bias you couldn't remove is what makes the rest of the numbers credible.
Checklist
Learn more
- Pew Research Center, Writing Survey Questions: wording, order, acquiescence and social desirability, with the experiments
- Jeff Sauro, 9 Biases That Affect Survey Responses (MeasuringU, 2016)
- Jon A. Krosnick, Response strategies for coping with the cognitive demands of attitude measures in surveys (Applied Cognitive Psychology, 1991)
- Caroline Jarrett, The Survey Octopus (Effortmark, free slides)
- Maddie Brown, Should You Run a Survey? (NN/g, 2024)
Books, courses and videos to go deeper
- Surveys That Work: A Practical Guide for Designing and Running Better SurveysCaroline JarrettBook · Rosenfeld Media, 2021 (Amazon)
- The Psychology of Survey ResponseRoger Tourangeau, Lance J. Rips and Kenneth RasinskiBook · Cambridge University Press, 2000 (Amazon)
Book links go to Amazon. As an Amazon Associate I earn from qualifying purchases, at no extra cost to you.
Related
Writing survey questions · Sample size and margin of error · Screeners and sampling · How people think · Write and pilot a survey