Marcus Andrey Study
Ler em português Library

UX Research · Step 5 · 8 min read

Bias in surveys

A survey can be wrong in two ways: it can ask the wrong people, or it can ask the right people in a way that bends their answers. The main biases by name, real experiments that show how big they are, and what you can do about each one.

A biased survey doesn't look biased. It looks like a clean chart. That's what makes bias dangerous: a percentage with a decimal point feels like a fact, whoever answered and however the question was put. You can't remove every bias. You can know their names, design against the big ones, and say honestly which ones are left.

What you'll be able to do after reading:

  • Tell errors about who answers from errors about how they answer
  • Name ten common survey biases and what each one does
  • Show with real numbers how much wording and order can move results
  • Reduce each bias in the design, and report the ones you couldn't

1. Two ways to be wrong

Survey methodologists group everything that can go wrong under one name, Total Survey Error: the idea is to consider all the errors together and make choices that keep the total low (Caroline Jarrett, Effortmark). Jarrett's Survey Octopus maps those errors onto the steps of a survey; for a designer, two big families are enough to start:

  • Who answers. The people who could be asked, were asked and chose to answer may not be the people you want to know about. Pew gives a plain example: people without internet access can't take online surveys (Pew, Understanding the margin of error, 2016).
  • How they answer. The right people, asked in a way that bends their answer: the wording, the order, the scale, who's asking.

More responses fix neither. Ten thousand answers from the wrong people, or to a leading question, are ten thousand wrong answers, with a smaller margin of error. As Jeff Sauro puts it, it's much better to ask a few of the right people than a lot of the wrong ones (MeasuringU, 2010).


2. The catalogue

BiasWhat happensIn the bill-splitting surveyHow to reduce it
CoverageThe people you can reach aren't the people you want to know aboutAn in-app survey never reaches people who gave up on the appName the group first; recruit former users too, by e-mail
Non-responseThe people who answer differ from the ones who don'tFans answer; people with a bad settle-up ignore itKeep it short, send one reminder, compare who answered with who you invited
AcquiescenceA tendency to agree with any statement"Reminders are helpful. Agree or disagree?"Offer a choice between two statements instead of agree or disagree
Social desirabilityAnswers that look good instead of answers that are trueFew admit they never paid a friend backAnonymous and self-completed; word it so the honest answer is normal
Question orderEarlier questions change how later ones are readAsking about a fight over money, then about overall satisfactionGeneral before specific; keep the same order when you track over time
Option orderOn screen, people pick options near the top more often"Message" always first in the list of reminder methodsShuffle lists with no natural order; keep scales in order
WordingSmall word changes move the answers"Nagging reminders" against "reminders"Plain, neutral words; test two versions on two halves of the sample
SponsorKnowing who asks colours the answersThe app's logo and "help us improve" in the invitationA neutral invitation; say who you are, without selling
MemoryPeople misremember when and how often things happened"How many times this year did you settle up?"A short, recent window: "in the past 7 days"
SatisficingTired respondents give the first acceptable answerThe same column ticked down a 15-row gridFewer questions, fewer grids, and don't make everything required
Ten biases worth knowing by name. The first two are about who answers; the other eight are about how they answer.Diagram: Marcus Andrey, CC BY 4.0. Sources: Pew Research Center, Writing Survey Questions; NN/g (Rosala, 2020; Brown, 2023); Jeff Sauro, 9 Biases That Affect Survey Responses (MeasuringU, 2016); Krosnick (1991).

You don't need to memorise the table; you need to recognise the pattern when you see it. A few deserve a closer look.

Acquiescence. Pew reports that less educated and less informed respondents are more likely to agree with agree-disagree statements, and more so when an interviewer is present. Their fix is to offer a choice between alternative statements instead (Pew Research Center).

Social desirability. People understate drinking, drug use and tax evasion, and overstate church attendance, charity and voting. Pew words its voting question to make "no" easy: "did things come up that kept you from voting, or did you happen to vote?" Self-completed web and paper surveys show less of this bias than phone or face-to-face ones. NN/g adds that asking for names and other identifiers increases it (NN/g, Rating Scales in UX Research, 2020).

Satisficing. When answering takes more effort than people want to give, they settle for an answer that seems good enough instead of the best one. Jon Krosnick described this response strategy in 1991, and it's why long surveys and big grids produce the same column ticked all the way down (Krosnick, Applied Cognitive Psychology, 1991).

Sponsor and memory. Knowing who sent the survey colours the answers, especially about brands. And people misplace events in time, remembering them as earlier or more recent than they were, which Sauro calls telescoping (MeasuringU, 9 Biases That Affect Survey Responses, 2016).


3. How big these effects are

Wording: favour military action in Iraq (January 2003)

"…to end Saddam Hussein's rule"68%
"…even if it meant that U.S. forces might suffer thousands of casualties"43%

Order: dissatisfied with the way things are going (December 2008)

Asked right after presidential approval88%
Asked without that question before it78%

Open or closed: the economy as the issue that mattered most (2008)

Closed: "the economy" offered in a list of five58%
Open: people answered in their own words35%
Same topic, same kind of people, different answers. Each pair compares two versions of one question in Pew's surveys: only the wording, the order or the format changed.Data: Pew Research Center, Writing Survey Questions. Chart: Marcus Andrey, CC BY 4.0.

These aren't small effects at the edge of statistics. Pew tests them by writing two versions of a question and giving each to a random half of the sample: because the halves are alike, any difference comes from the wording (Pew Research Center). Some of the results Pew reports:

  • Wording. In 2003, 68% favoured military action in Iraq "to end Saddam Hussein's rule". Adding "even if it meant that U.S. forces might suffer thousands of casualties" dropped it to 43%.
  • Emotional words. In 2005, 51% favoured letting doctors "give terminally ill patients the means to end their lives"; 44% favoured letting them "assist terminally ill patients in committing suicide". Same policy, different words.
  • Order. In 2008, 88% said they were dissatisfied with the way things were going in the country when the question came right after one about the president's approval; 78% without it.
  • Open or closed. 58% against 35% for "the economy", depending on whether it was offered in a list (see Writing survey questions).

Your product survey won't be about war. But "nagging reminders" against "reminders", or a satisfaction question right after a question about a failed payment, work the same way.


4. Order: what comes before changes what comes after

Pew describes two kinds of order effect. In a contrast effect, answers move apart because of the order; in an assimilation effect, they move closer together. A specific question asked before a general one, like happiness with your marriage before happiness overall, tends to produce contrast: people leave the specific part out of the general answer (Pew Research Center).

Options have an order too. Pew notes that in self-completed surveys people tend to pick options near the top of a list (a primacy effect), and on the phone, options heard last (recency). What to do:

  1. General before specific. Ask overall satisfaction before the questions about one bad experience.
  2. Shuffle lists that have no natural order, so every option gets the top spot equally often. Pew notes this doesn't remove the effect; it spreads it evenly.
  3. Keep scales in order, but you can reverse a scale for half the respondents, as Pew does on some questions.
  4. Keep the order fixed when you compare over time, or the change you see may be a change in context.
  5. Put sensitive and demographic questions near the end, unless you need them to screen people. Sauro also notes that questions which bring back bad memories can sour the answers that follow (MeasuringU, 2016).

5. Who answered?

The two "who" biases are the ones teams forget, because the data looks fine.

  • Coverage. An in-app survey reaches only people who still open the app. If your question is why people leave, the people who left can't answer it.
  • Non-response. Of the people you invite, the ones who answer are different: often more engaged, happier, or angrier. NN/g quotes researchers who call surveys the hardest method to do well and the easiest to launch, and advises starting one only with enough time to do it with rigour (NN/g, Should You Run a Survey?, 2024).

Three habits help:

  1. Write down the population before you write a question: "people who added at least one shared expense in the last 90 days".
  2. Compare respondents with invitees on what you already know, such as plan, country, months since sign-up. If the respondents are mostly long-time users, say so.
  3. Recruit the missing group on purpose, for example former users by e-mail, instead of hoping they'll turn up in the in-app survey.

6. Common mistakes

  • Believing a big sample cancels bias. It shrinks the margin of error, not the bias.
  • Agree-disagree statements for everything.
  • A satisfaction question placed right after the questions about problems.
  • Asking for name and e-mail at the start of a survey about sensitive behaviour.
  • Reporting "78% of users" when it's 78% of the people who chose to answer.
  • Changing one word in a tracked question, then celebrating the trend.

7. Vocabulary

Term Meaning
Total Survey Error All the errors a survey can have, considered together
Coverage error The people you can reach aren't the population you care about
Non-response bias The people who answer differ from those who don't
Acquiescence A tendency to agree with statements
Social desirability Giving the answer that looks good instead of the true one
Satisficing Giving a good-enough answer to save effort
Order effect Earlier questions changing how later ones are answered
Primacy, recency Picking options at the start, or at the end, of a list
Split-ballot test Two versions of a question, each given to a random half

8. Try it (no tools needed)

  1. Take a survey you've received recently.
  2. Who could not have answered it? Write down one group.
  3. Find one question whose answer might change if the question before it were different.

Reflection questions: If the survey's results were published tomorrow, which bias would you mention in the footnote? Would anyone in your team read the footnote?


9. If you already practise

  • Add a "who answered" table to every survey report: invited, responded, and how they compare on what you know.
  • When a question matters, run a split-ballot test of two wordings in the pilot, the way Pew does at scale.
  • Randomise unordered lists by default in your survey tool, and turn it off on purpose, not by accident.
  • Keep a limitations section in the report. Naming the bias you couldn't remove is what makes the rest of the numbers credible.

Checklist

Learn more

Books, courses and videos to go deeper

Book links go to Amazon. As an Amazon Associate I earn from qualifying purchases, at no extra cost to you.

Writing survey questions · Sample size and margin of error · Screeners and sampling · How people think · Write and pilot a survey

Saved only on this device.

↑ ↓ to move · Enter to open · Esc to close