UX Research · Step 4 · 7 min read
Planning a usability test
A usability test is simple to run and easy to waste. How to start from the decision the test will inform, choose between qualitative and quantitative, moderated and unmoderated, in person and remote, write a one-page plan, and pilot it so your first participant isn't also your rehearsal.
A usability test answers one kind of question very well: can these people do this thing with this design, and if not, why not? It's the most direct evidence a designer can get, and it's cheap. That's also why it's easy to waste: tested too late to change anything, with the wrong people, on tasks that give away the answer, with nobody from the team watching. A good test is decided before the first participant arrives, on one page.
What you'll be able to do after reading:
- Start a test from the decision it will inform
- Choose qualitative or quantitative, moderated or unmoderated, in person or remote
- Write a one-page test plan
- Pilot the test and get the team to watch
1. What a usability test is
In Usability testing with five people you ran the simplest version. The basic parts are always the same: a facilitator (or moderator) gives tasks to a participant, then watches what they do and listens to what they say (NN/g, Moran, Usability Testing 101, 2019). Nielsen's three-step version of a think-aloud test is even shorter: recruit representative users, give them representative tasks, and shut up and let them do the talking (NN/g, Nielsen, Thinking Aloud, 2012).
What a test isn't: a way to ask whether people like a design, a demo, or a vote between two options. It's observation of behaviour on a real task. And it's different from an interview: in an interview you want their story; in a test you want their actions, and you talk as little as possible (see Moderating and thinking aloud).
2. Start from the decision
Before choosing screens or people, write down what the test will decide. "Do we build this flow as designed, or change it before the sprint?" is a decision. "Get feedback on the new design" isn't. Then write the two or three questions the test has to answer to make that decision. Every task you write later should serve one of them, the same way interview questions serve research questions (see Research questions that lead to decisions).
Timing matters as much as wording. A test is worth most when the answer can still change the design: on a paper sketch, a clickable prototype, or a live product before the next release. Thinking aloud works at any stage, from early paper prototypes to finished systems (Nielsen, 2012).
3. Three choices
Qualitative or quantitative? Qualitative testing collects findings and their causes, and is best for discovering problems. Quantitative testing collects metrics such as task success and time on task (Moran, 2019). They need very different numbers of people: a handful for qualitative, around 40 for most quantitative studies (NN/g, Budiu and Moran, 2021). Most design teams need qualitative most of the time. The details are in How many participants and Measuring usability.
Moderated or unmoderated? In a moderated test, a facilitator is there (in the room or on a call), gives instructions, watches in real time and asks follow-up questions. In an unmoderated test, software gives the tasks, records the session and may ask fixed questions afterwards; no researcher attends. Unmoderated studies are faster and can reach dozens of people at once, but nobody can help a participant recover from a prototype's limits or ask why. NN/g says they work best for live sites and apps or highly functional prototypes (NN/g, Whitenton, Unmoderated User Tests, 2019).
In person or remote? NN/g recommends in person whenever possible, because it's easier to read body language and know when to probe. But when budget or time rule it out, remote testing is better than skipping the test (NN/g, Schade, Remote Usability Tests, 2013). Remote also lets you see people on their own devices and setups, which is a finding in itself.
4. The one-page plan
Test plan: adding a shared bill, round 1
Decision
Do we build the "add a bill" flow as designed, or change it before the sprint starts?
Questions
- Can people add a bill and split it unevenly without help?
- Do they understand who owes what once it's added?
- Where do they hesitate, and what did they expect instead?
What we test
Figma prototype, phone size, shared by link. Four screens linked; everything else shows "not in this prototype".
Participants
5 people who share bills with at least one other adult and split one in the last month. None from our company. Screener in the study folder.
Tasks
- Explore: look around and tell me what you think this is for.
- Add last month's internet bill, shared by three.
- One flatmate was away for half the month: make the split fair.
- Find out how much you're owed in total.
Success
Task done without help, and the amounts on screen are right. Note every hesitation and every wrong path.
Session45 min
Welcome and consent 5 · think-aloud practice 3 · tasks 25 · questions 10 · thanks 2.
Roles
Moderator: designer. Note-taker: researcher. Observers (muted, cameras off): PM and one developer.
Each part has a reason:
- Decision and questions keep the test small. If a question doesn't help the decision, it belongs in another study.
- What we test says exactly which screens work, so nobody is surprised by a dead end. In a prototype, put a polite "not in this prototype" screen on every unlinked path.
- Participants are people who really have the goal, not colleagues who know the product. Recruit them with a screener (Screeners and sampling, Recruiting participants).
- Tasks go from exploratory to specific, the funnel again (Tasks and scenarios).
- Success is defined before the session, per task, so you aren't deciding afterwards what counted.
- Session timing keeps the tasks inside the time. If you add a short interview before the tasks, plan a longer session: NN/g gives 90 minutes instead of 60 as an example (NN/g, User Interviews 101, 2023).
- Roles keep the moderator free to watch. A note-taker writes; observers stay silent.
5. Pilot it
A pilot test is a dress rehearsal: one or two sessions before the real study, to test the study, not the design. It checks that tasks are understood, that the session fits the time, and that the prototype, consent forms and payments are ready (NN/g, Schade, Pilot Testing, 2015). It matters most for unmoderated tests, where instructions must stand on their own, and for high-stakes studies, where you won't get a second try.
A colleague from another team is often enough as a pilot participant. Fix what confused them, and the first real participant gets the test you meant to run.
6. Get the team to watch
Nielsen's point is worth repeating: the most hard-boiled developers and tight-fisted executives usually "soften up" once they see first-hand how customers think about their work, and sitting in on a few sessions doesn't take much of their time (Nielsen, 2012). Invite them in the plan, give them a simple observation sheet (what did they try, where did they hesitate, what did they say), and meet for 30 minutes after the last session to agree the top problems. Problems the team saw together get fixed faster than problems in a report.
7. Common mistakes
- Testing when the design can no longer change.
- "Get feedback" as the goal, with no decision behind it.
- Testing with colleagues, friends or people who don't have the goal.
- Running unmoderated tests on rough prototypes that need a person to explain the gaps.
- No pilot, so the first session tests the study instead of the design.
- Writing the report before anyone on the team has watched a session.
8. Vocabulary
| Term | Meaning |
|---|---|
| Facilitator / moderator | The person who runs the session and gives the tasks |
| Qualitative test | Finds problems and their causes, with a few participants |
| Quantitative test | Measures how well a design performs, with many participants |
| Moderated | A facilitator is present, in person or remotely, in real time |
| Unmoderated | Software runs the session; the researcher reviews recordings later |
| Pilot test | A rehearsal session that tests the study itself |
| Observer | A team member who watches silently and takes notes |
9. Try it (no tools needed)
- Pick a design you're working on, or any app you use.
- Write one decision a test could inform, and two questions it would need to answer.
- Fill in the plan headings on one page: participants, three tasks, success, timing, roles.
Reflection questions: Could the answer still change the design? Who on your team would most benefit from watching a session?
10. If you already practise
- Keep a test plan template and a standing slot: one small test every sprint or two beats one big study a quarter.
- Pilot with someone from another team: they're outside enough to be confused by the same things.
- For unmoderated tests, pilot twice: once with a colleague, once with a real participant from the panel.
- Steve Krug's Rocket Surgery Made Easy is a complete do-it-yourself system for small, regular tests with the team watching.
Checklist
Learn more
- NN/g: Usability (User) Testing 101 (Moran, 2019) · Thinking Aloud: The #1 Usability Tool (Nielsen, 2012) · Remote Usability Tests: Moderated and Unmoderated (Schade, 2013) · Unmoderated User Tests: How and Why to Do Them (Whitenton, 2019) · Pilot Testing (Schade, 2015)
Books, courses and videos to go deeper
- Rocket Surgery Made EasySteve KrugBook · New Riders, 2010 (Amazon)
- Usability (User) Testing 101NN/gfree, article (external site)
Book links go to Amazon. As an Amazon Associate I earn from qualifying purchases, at no extra cost to you.
Related
Usability testing with five people · Test with three people · Choosing a research method · Writing a research plan · Prototype fidelity · Heuristic evaluation · Usability task scenarios