UX Research · Labs · 3 min read
Write and pilot a survey
A guided lab: write a short survey tied to your research questions, pilot it with five people thinking aloud, time it, run a small online pilot, and fix every question that was misread. About three hours, with any survey tool and five patient people.
Every survey question is a small interface, and nobody tests it. Once a survey goes out, you can't ask a respondent what they meant. A pilot is the one chance to watch people read your questions and find out that "last month" meant four different things. It takes an afternoon and saves the whole dataset.
What you'll do:
- Write a survey of eight questions or fewer, each tied to a research question
- Pilot it with five people thinking aloud
- Time it, and keep the median as a baseline
- Run a small online pilot and look at the raw answers
- Fix what was misread, and note what you changed
You'll need: a free survey tool, three research questions (from Write a research plan or your own), five people who match your audience, about three hours.
1. Write it short, and tie every question to a reason
Start from your research questions, not a blank form. For each survey question, write next to it which research question it answers and what you'll do with the answer. If you can't, cut it. Follow Writing survey questions for wording and Bias in surveys for order and scales. For this lab, keep it to eight questions or fewer, at least one of them open.
2. Pilot it with people thinking aloud
Sit with each person (in person or on a call) while they fill it in, and ask them to think aloud. Don't help; write down where they hesitate, reread or answer a different question from the one you asked. NN/g's case study of a single survey question went through four pilot rounds before launch; each round revealed a new misreading, and the authors suggest that "testing 5–10 users for each version of your pilot should work fine", with at least one round with your real audience (Feifei Liu, NN/g, 2019).
After each person, ask two questions: "What did you think question 3 was asking?" and "Was there an answer you wanted to give and couldn't?"
3. Time it, and keep the number
Note how long each person took, without the think-aloud if you can (ask one or two to do it silently). Keep the median. You'll use it twice: to tell respondents honestly how long it takes, and to spot bots later, since NN/g advises flagging completions far quicker than your pilot median (Rachel Banawa, NN/g, 2026).
4. Check it with a pilot checklist
Pilot checklist: household bills survey, v1
Questions
- Each question is read the way we meant itQ2 "How many bills did you split last month?": 3 of 5 counted shared groceries, 2 didn't. v2 lists what counts.
- Every option a person needs is thereYes, after adding "We don't split, one person pays" in the pilot's first session.
- No question leads to an answerQ5 "How frustrating is splitting bills?" assumed frustration. v2: "How easy or difficult…".
- The open question gets stories, not one wordYes: "Tell us about the last time a split was questioned" got 2 to 4 sentences each.
Flow and time
- Fits the time we promiseMedian 6 min 40 s; the invitation says "about 7 minutes".
- Skip logic sends people to the right placePeople who don't split were still asked how they split. Fixed the branch.
Messages and data
- Invitation, consent text and thank-you page make sense aloneYes, read cold by 2 people who weren't in the pilot.
- The export answers each research questionYes, mapped column by column to RQ1–RQ3.
- We know what a suspicious response looks likeFlag under 2 minutes, repeated open answers, batches.
NN/g calls a pilot "a dress rehearsal for the study": it tests the study, not the design, and for unmoderated studies it should also test every message people receive, since nobody can ask you what you meant (Amy Schade, NN/g, 2015). Go through the checklist with your notes from the five sessions.
5. Run a small online pilot, then read the raw data
Send the fixed version to 10 to 20 people the way the real survey will go out. Then open the raw export, not the summary charts:
- Can you answer each research question with these answers?
- Are open answers on topic, or are people answering something else?
- Does any option get nobody, or everybody?
- Did any response finish far below the median?
Fix, and write a short changelog: question, what was wrong, what you changed.
6. Check your work
Reflection questions: Which question did you think was clearest, and which did people misread most? What would you have concluded from the data if you hadn't piloted?
Related
Writing survey questions · Bias in surveys · Sample size and margin of error · AI in UX research, carefully