Marcus Andrey Study
Ler em português Library

UX Research · Step 5 · 9 min read

Writing survey questions

A survey question works when every respondent reads it the same way and can answer it truthfully. What a survey is for, the four steps people go through to answer, the classic mistakes and their rewrites, open and closed questions, rating scales, and why you pilot before you send.

Response cards sorted into five piles on a base, one pile for each answer, so the piles stand as the bars of a chart. Point at a bar and its cards lift apart, so you can count them one by one. That's this step in one gesture: every bar in a survey chart is made of individual people's answers. This step has four lessons: writing survey questions, bias in surveys, sample size and margin of error, and analytics and A/B tests for designers.Illustration: Marcus Andrey, drawn with Hairline (MIT, Lucas Marques).

In an interview you can see confusion on a face and ask again. In a survey you can't. The question goes out alone, hundreds of times, and every person reads it without you. If some read "easy" and others read "quick", you get a number that means nothing, and it looks exactly like a number that means something. So survey writing is mostly editing: taking a question you understand and making it one that everybody understands the same way.

What you'll be able to do after reading:

  • Decide whether a survey is the right method, starting from the decision
  • Explain the four steps a person goes through to answer a question
  • Spot and fix the classic question mistakes
  • Choose between open and closed questions, and build a balanced rating scale

1. A survey is for a number you'll use

Caroline Jarrett, who wrote the book on surveys for UX people, defines it like this: a survey is "a process of asking questions that are answered by a sample of a defined group of people to get numbers that you can use to make decisions" (Jarrett, excerpt from Surveys That Work). Every part of that sentence is a test:

  • A defined group of people. Not "users". Who, exactly? People who split a bill in the app last month? People who stopped?
  • Numbers. If what you need is "why", that's an interview (see Short user interviews). A survey answers "how many".
  • Decisions. Jarrett's first challenge for every draft question is "What decision will you make based on the answers?" If there's none, the question goes on a list for another time.

She then asks you to pick your Most Crucial Question, the one you'd keep if you could keep only one, and to finish this sentence: "We need to ask ___ so that we can decide ___." If you can't fill both blanks, you're not ready to write questions yet. It's the same idea as Research questions that lead to decisions, applied to a questionnaire.

NN/g adds a practical limit: surveys are best for quantitative attitudinal data, what people think and report, and poor at behaviour you could observe. Asking "would this button stand out?" in a survey gets you guesses; a usability test or an A/B test would show you (NN/g, Writing Good Survey Questions, 2023).


2. How people answer a question

  1. 1Understand

    Work out what the question means and what kind of answer it wants.

    Breaks on jargon, two questions in one
  2. 2Remember

    Find the facts or feelings it asks about.

    Breaks on "a typical week", long ago
  3. 3Judge

    Put those memories together into one answer.

    Breaks on predictions, leading context
  4. 4Answer

    Fit that answer into your options, and decide whether to say it.

    Breaks on missing options, sensitive topics
Every answer goes through four steps in the respondent's head, and a bad question can break any of them.Diagram: Marcus Andrey, CC BY 4.0, after the steps in Tourangeau, Rips and Rasinski, The Psychology of Survey Response (Cambridge University Press, 2000).

Survey methodologists describe answering as a short chain of mental work: understand the question, remember what's relevant, put it together into a judgement, and map that judgement onto the answer options, deciding along the way whether to say it honestly. The chapters of Tourangeau, Rips and Rasinski's book follow exactly that chain (Cambridge University Press, The Psychology of Survey Response, 2000).

The chain is useful because every classic mistake breaks one link. Jargon breaks understanding. "In a typical week" breaks memory, because nobody stores an average. A leading sentence breaks judgement. Options that don't fit break the last step. When a question feels wrong, ask which link it breaks.


3. The classic mistakes, and their rewrites

  1. Two questions in one"Easy and quick" can't share one answer
    Before

    How easy and quick was it to split the bill?

    After

    How easy or difficult was it to split the bill?

    Then, separately: about how long did it take?

  2. A predictionPeople guess their future badly
    Before

    Would you use automatic reminders?

    • Yes
    • No
    After

    The last time someone owed you money for a shared bill, how did you remind them?

    • Message
    • In person
    • In the app
    • I didn't remind them
    • Other
  3. An "average week"Nobody remembers an average
    Before

    How often do you add an expense in a typical week?

    After

    About how many expenses did you add in the past 7 days?

    • None
    • 1–2
    • 3–5
    • 6 or more
    • I don't remember
  4. A leading startThe question asks for a good mark
    Before

    We work hard to make settling up painless. How satisfied are you?

    After

    How would you rate your last settle-up?

    • Very good
    • Good
    • Neither
    • Poor
    • Very poor
  5. Options that overlap and leave gapsA 25-year-old fits twice; a 60-year-old fits nowhere
    Before

    How old are you?

    • 18–25
    • 25–35
    • 35–50
    After

    Do you need age at all? If you do:

    • 18–24
    • 25–34
    • 35–49
    • 50 or over
    • Prefer not to say
  6. An unbalanced scaleThree good options, one bad
    Before
    • Excellent
    • Very good
    • Good
    • Poor
    After
    • Very good
    • Good
    • Neither good nor poor
    • Poor
    • Very poor
Six questions from a draft survey for a bill-splitting app, before and after a review. Each fix makes the question one that every respondent reads the same way and can answer truthfully.Diagram: Marcus Andrey, CC BY 4.0. Rules from Pew Research Center, Writing Survey Questions, and NN/g, Writing Good Survey Questions: 10 Best Practices (2023).

Each row of the sheet is one rule from Pew Research Center's guide to writing survey questions and NN/g's ten best practices:

  • One question at a time. "Easy and quick" are two things. A double-barrelled question forces people to answer one half, or average both, and you can't tell which.
  • Ask about the past, not the future. People are poor predictors of their own behaviour. "Would you use it?" gets a polite yes.
  • Ask about a recent, specific window. NN/g's rewrite of "in an average week" is "approximately how many times… in the past 7 days?", with ranges, because exact counts are hard and push people to drop out.
  • No context that asks for a good mark. NN/g's example opens with "We are committed to achieving a 5-star satisfaction rating" before asking for one. Remove the plea and just ask.
  • Options that cover everything and never overlap. Pew asks for lists that are exhaustive and mutually exclusive. Check number ranges twice.
  • Balanced scales. As many negative options as positive ones around the middle, or you nudge people towards the side with more options.
  • An honest way out. "I don't know", "Not applicable" or "Prefer not to say", kept as separate options, because they mean different things.

Two more that don't fit in a picture. Plain words: Pew warns against double negatives and unfamiliar abbreviations. And don't make everything required: someone who can't answer a required question either picks at random, which you'll never detect, or leaves the survey (NN/g, 2023).


4. Open or closed

A closed question gives options; an open one lets people answer in their own words. The choice changes the answers. After the 2008 US election, Pew asked which issue mattered most to people's vote: when "the economy" was offered in a list, 58% chose it; when people answered in their own words, 35% said it. In the open version, 43% gave an answer that wasn't on the closed list at all (Pew Research Center).

So the list is never neutral: it reminds people of some answers and hides others. A good routine, which Pew describes too, is to ask an open question first, in a pilot or in interviews, and build the closed options from what people actually said.

In practice:

  • Use mostly closed questions, because the point of a survey is counting. If your draft is mostly open questions, NN/g suggests another method may suit you better.
  • Keep lists short: Pew suggests four or five options for opinion questions, more only for facts like religion or country.
  • Prefer one choice per question to "select all that apply": Pew found forced-choice questions give more accurate answers and generally avoids select-all lists.
  • End with one optional open question, such as "Is there anything else you'd like to tell us?". People often arrive with something to say, and this is where it goes.

5. Rating scales

Two kinds of scale do most of the work in UX surveys (NN/g, Rating Scales in UX Research, 2020):

  • A Likert item asks how much you agree with a statement, usually on five labelled points, from strongly disagree to strongly agree. The System Usability Scale is built from ten of them.
  • A semantic differential puts two opposite words at the ends of a scale, usually seven points, with the middle unlabelled. The Single Ease Question, "very difficult" to "very easy", is one.

Agree-disagree items have a known weakness: some people tend to agree with whatever they're shown. Pew calls it acquiescence bias and prefers offering a choice between two statements. Flipping half the statements to negative wording is a common fix, but Sauro and Lewis found it confuses both respondents and the researchers scoring the answers (CHI 2011). The next lesson, Bias in surveys, goes further.

Three rules cover most cases: use a standardised questionnaire when one exists for what you're measuring, instead of inventing your own; use the usual labels ("strongly agree", "agree", "neither agree nor disagree"…), and add "Not applicable" where a question may not apply, so you can tell the neutral from the irrelevant (NN/g, 2020).


6. Pilot it before you send it

Pew calls pretesting "an essential step", especially for new questions (Pew). When NN/g replicated a 1998 survey, they piloted one open question in four rounds. In the first round, 4 of the 11 pilot participants said the question was too general and they weren't sure what was wanted; each round changed the wording again (NN/g, Iterative Design of a Survey Question, 2019).

A pilot doesn't need to be big:

  1. Ask three to five people from the real audience to fill in the survey while thinking aloud, as in Moderating and thinking aloud.
  2. After each question, ask: "What did you think this question was asking?" Write down every answer that's different from what you meant.
  3. Fix, and pilot the fixed version once more.
  4. Then send it to a small slice of the list and check the numbers look sane before sending it to everyone.

7. Common mistakes

  • Starting from "what would be interesting to know" instead of "what will we decide".
  • Asking people to predict what they'll do, then treating the answer as a forecast.
  • Questions that ask two things at once.
  • Rating scales with more good options than bad ones.
  • Every question required.
  • Demographic questions first, "just in case". Pew advises against opening with them, and NN/g asks whether you need them at all.
  • Sending without a pilot, because "it's only a short survey".

8. Vocabulary

Term Meaning
Survey The whole process: goals, sample, questions, fieldwork, analysis
Questionnaire The set of questions people actually see
Double-barrelled question One question that asks about two things
Closed question A question with options to choose from
Open question A question people answer in their own words
Likert item A statement rated from strongly disagree to strongly agree
Semantic differential A scale between two opposite words
Balanced scale As many negative as positive options, around a midpoint
Pilot A small trial run of the survey before the real one

9. Try it (no tools needed)

  1. Find a survey in your inbox, from an app, a shop or your employer.
  2. For each question, say which of the four steps it makes hard: understand, remember, judge or answer.
  3. Rewrite the worst one, and read it to someone. Ask them what they think it's asking.

Reflection questions: Which decision was that survey meant to support? Could you tell from the questions?


10. If you already practise

  • Write the "We need to ask ___ so that we can decide ___" sentence at the top of every survey brief, and make stakeholders sign it before they add questions.
  • Keep a team bank of tested questions, with their exact wording and the date they were piloted. Pew keeps wording identical when it tracks over time; do the same, or your trend is a trend in wording.
  • Run cognitive pilots with three people for every new question. It costs an hour and saves a fieldwork.

Checklist

Learn more

Books, courses and videos to go deeper

Book links go to Amazon. As an Amazon Associate I earn from qualifying purchases, at no extra cost to you.

Bias in surveys · Asking good questions · Measuring usability · Forms · Write and pilot a survey

Saved only on this device.

↑ ↓ to move · Enter to open · Esc to close