Marcus Andrey Study
Ler em português Library

UX Research · Step 2 · 8 min read

Screeners and sampling

A screener decides who gets into your study, and people who want the incentive will try to guess the right answers. How to write screener questions that can't be gamed, how qualitative sampling works, what saturation really means, and how quotas keep a small sample from being all the same person.

A screener is a small survey with one job: to let the right people in and keep the wrong ones out, even when the wrong ones want in. Ask "Are you a freelancer?" and every person who'd like the voucher is suddenly a freelancer. Ask about what they actually did last month, among several plausible answers, and the study fills with the people it was meant for. Then the question is how many, and of which kinds.

What you'll be able to do after reading:

  • Write inclusion and exclusion criteria that come from the research questions
  • Write screener questions that are hard to game
  • Explain purposive sampling and saturation, and why there's no magic number
  • Use quotas so a small sample covers the people who matter

1. What a screener is

NN/g defines a screener as "a set of questions, typically delivered verbally or via written questionnaire, used to establish that a prospective participant is a good fit for a given research project" (Brown, Screening Participants for User-Research Studies, NN/g, 2024). It does more than filter: it improves data quality, saves sessions, and reduces bias, for example from panels full of tech-savvy volunteers.

The screener comes from the recruitment brief (see Recruiting participants), and the brief comes from the research questions. If a screener question doesn't trace back to a research question, cut it.


2. Inclusion and exclusion

Start with two lists (Brown, NN/g, 2024):

  • Inclusion criteria: what makes someone a good fit. Mostly behaviours: what they do, how often, with which tools.
  • Exclusion criteria: what rules them out. Commonly: people who work in UX, marketing or IT for interface studies, people who took part recently, and "professional testers" who make a living from studies.

Then resist overscreening. NN/g warns that matching every trait of your ideal user produces "an overly restrictive screener that nobody will be able to pass". Screen only for what's likely to affect your research questions.


3. Questions that can't be gamed

WeakAre you a freelancer?
  • Yes (accept)
  • No (reject)

Anyone who wants the voucher knows which answer gets in.

BetterWhich best describes your work right now?
  • Employed full-time (reject)
  • Freelance or self-employed (continue)
  • I run a business with employees (reject)
  • Other: ______

Several plausible answers hide the one you want.

WeakWould you use recurring invoices?

A prediction, and it gives away what the study is about.

BetterIn the last 3 months, how many clients did you invoice more than once?
  • 0 · 1 to 2 · 3 to 5 · 6 or more (quota)

Past behaviour, in ranges, with a time frame.

WeakHow satisfied are you with our onboarding UX?

Jargon, and only customers can answer it.

BetterTell us about the last invoice you sent: who was it for, and how did you send it?

Open. Two lines show the context and how well they'll talk in a session.

WeakDo you work in tech? Yes · No

Easy to dodge, and "tech" means something different to everyone.

BetterDo you or anyone in your household work in any of these?
  • Market research · Design · Software (reject)
  • None of these (continue)

Concrete fields, with a neutral way out.

Quotas for 6 sessions:
  • 4 who invoiced one client once, 2 who invoice the same clients every month
  • No more than 2 from the same trade; at least 2 who don't use a computer every day for work
  • Ask about access needs after selection, so you can adapt the session, never to filter people out
Four screener questions, weak and better. The better versions ask about past behaviour, hide the answer you want among plausible ones, and use plain words. The quotas at the end keep the sample from filling up with one kind of person.Diagram: Marcus Andrey, CC BY 4.0. Practices from Maddie Brown, Screening Participants for User-Research Studies (NN/g, 2024).

NN/g's best practices, all visible in the figure (Brown, NN/g, 2024):

  • Ask about past behaviour, not predictions. "Users are notoriously unreliable at predicting their future behaviors", sometimes from "an attempt to lie their way into a paid research opportunity".
  • Avoid yes/no questions. Their intent is easy to guess. Use several plausible options, or a two-question series ("How many times did you shop online last month?" then "Which sites?").
  • Avoid leading questions, and funnel. Start broad and reveal the topic late, so people can't tailor their answers.
  • Include one or two open questions. They give you context, and a one-word answer may predict a one-word session.
  • Use plain words. NN/g's example swaps "ethnographic research for product ideation" for "observed or interviewed people in their real-life settings to get ideas for new products".
  • Pilot it. Try the screener on a few people to catch typos, ambiguous wording and broken skip logic.

A short phone call after the written screener (NN/g suggests 2 to 5 minutes) lets you ask the revealing questions in conversation and hear how someone talks, which matters if your study depends on them explaining their thinking.

The Screener questionnaire card puts these rules on one page, with a worked example and a template to copy.


4. How qualitative sampling works

Interviews and usability tests don't use random samples. They use purposive samples: people chosen on purpose because they can inform the research questions. A study of purposive sampling calls it "the most commonly used form of nonprobabilistic sampling" (Guest, Bunce and Johnson, How Many Interviews Are Enough?, Field Methods, 2006).

This has a consequence people forget: a qualitative sample can't tell you how common something is. If 4 of your 6 participants copy invoices by hand, you've learned that the behaviour exists and why. You haven't learned that 67% of freelancers do it. For "how many", you need a quantitative method and a proper sample size (see Sample size and margin of error).


5. How many, and saturation

There's no golden number. NN/g says so directly for interviews, and explains the idea that replaces it: saturation, "a point where themes emerging from the research are fleshed out enough such that conducting more interviews won't provide new insights that would alter those themes" (Rosala, How Many Participants for a UX Interview?, NN/g, 2021).

What the evidence says:

  • In the study by Guest, Bunce and Johnson, with sixty interviews, "saturation occurred within the first twelve interviews, although basic elements for metathemes were present as early as six" (Field Methods, 2006). That was one study, of women in two West African countries; more varied groups and broader questions may need more.
  • For usability testing, Nielsen and Landauer's model found that 5 users uncovered about 85% of the problems; with several distinct groups, Nielsen recommends 3 to 4 users per group for two groups, and 3 per group for three or more (Nielsen, Why You Only Need to Test with 5 Users, NN/g, 2000). That rule is about finding usability problems in one design, not about understanding needs.
  • NN/g warns against carrying the "5 users" rule over to interviews: for many exploratory studies, "5 participants are too few", while "5 per persona" can mean far more interviews than you need (Rosala, NN/g, 2021).

So: start small, analyse as you go, and stop when new sessions stop changing your themes. Plan for a number (six is a common start), and keep the option of two more.


6. Quotas: small but not all the same

With six people, one accident can tilt the whole sample: five from the same trade, all fluent with computers, all fans. Quotas prevent it. A quota says how many of each kind you need: "4 who stopped, 2 who stayed; no more than 2 from one trade."

Two kinds of diversity deserve a quota by default:

  • The ones that change the answer. People who stayed versus people who left; new versus experienced; different devices.
  • The ones that are usually missing. NN/g warns that testing "only with able-bodied and neurotypical people who are tech-savvy and literate" overlooks issues that affect a large part of the population (Brown, NN/g, 2024). GOV.UK goes further for public services: you need to include disabled participants (GOV.UK, Finding participants).

Ask about access needs to adapt the session, never to filter people out.


7. Common mistakes

  • Yes/no screener questions that announce the right answer.
  • Asking people what they would do instead of what they did.
  • Overscreening until nobody qualifies.
  • Reporting percentages from a qualitative sample.
  • Using "5 users" for every kind of study.
  • No quotas, so the sample is six versions of the same person.

8. Vocabulary

Term Meaning
Screener A short questionnaire that decides who joins the study
Inclusion / exclusion criteria The traits that qualify someone, and the ones that rule them out
Funnel technique Starting with broad questions and revealing the topic late
Purposive sample People chosen on purpose because they can inform the research questions
Saturation The point where new sessions stop changing the themes
Quota How many participants of each kind the sample needs

9. Try it (no tools needed)

  1. Write the worst possible screener question for your study: yes/no, leading, full of jargon.
  2. Rewrite it as a past-behaviour question with four or five plausible options, and mark which answers continue.
  3. Write the quotas for six sessions on one line.

Reflection questions: If you were a participant who wanted the voucher, which of your questions could you game? Which group of people would your quotas still leave out?


10. If you already practise

  • Keep a bank of tested screener questions for your product's main behaviours. Reuse them, and track which ones let the wrong people through.
  • Add one open question to every screener and read the answers before you book: it's the cheapest quality check you have.
  • In readouts, say what your sample can and can't tell people: "6 purposive participants; this shows what happens and why, not how often."

Checklist

Learn more

Books, courses and videos to go deeper

Book links go to Amazon. As an Amazon Associate I earn from qualifying purchases, at no extra cost to you.

Recruiting participants · How many participants · Sample size and margin of error · Short user interviews · Usability testing with five people · Inclusive design · Screener questionnaire

Saved only on this device.

↑ ↓ to move · Enter to open · Esc to close