UX Research · Step 7 · 7 min read
AI in UX research, carefully
AI can save hours in research and quietly put things in your report that nobody said. The risks that matter most for a research team, from invented quotes and synthetic users to AI respondents inside your own survey data, and the checks that keep the evidence honest.
The danger isn't that AI gets research wrong. It's that it gets it wrong fluently. A transcript with one invented sentence looks exactly like a correct one. A synthetic user sounds like a customer. A bot's survey answer reads like a thoughtful person's. Research is the part of the product process whose whole job is to be true, so the checks have to be stronger here than anywhere else.
What you'll be able to do after reading:
- Name the six risks AI brings into research, and where each one enters
- Check quotes against the recording before they reach a report
- Explain to a stakeholder why synthetic users are not user research
- Spot AI respondents in a survey, and say what you removed and why
1. Start with the AI for UX lessons
This lesson assumes you know where AI helps across the research cycle. The AI for UX course covers that: AI in UX research goes stage by stage, Synthetic users and digital twins reviews the studies on AI respondents, and AI-assisted synthesis, step by step gives a method for analysis with AI that keeps the rigour. The short version, from NN/g: AI might "miss, misinterpret, or even manufacture insights" (Kate Moran and Maria Rosala, NN/g, 2024).
Here we go one step further, into the risks a research team has to own: the ones that put false evidence in front of people who will make decisions with it.
2. Six risks, and the check for each
| Risk | Where it enters | What it looks like | The check |
|---|---|---|---|
| Invented quotes | Transcription, summaries | A clean, quotable sentence nobody said | Hear every quote in the recording; add code and timestamp |
| Invented or missed findings | AI analysis | Frequent themes in, the important rare one out | Read sessions first; trace each insight to raw data |
| Synthetic users as evidence | Planning, "validation" | Agreeable personas, a dozen answers where people give fifty | Label it "not user research"; never in the findings |
| Bots in your data | Surveys, unmoderated tests | Fluent answers, fast completions, batches | Several signals together; document what you removed |
| Lost behaviour | Usability analysis from transcripts | What people said, not what they did | Watch the sessions |
| Participants' data | Uploading recordings and transcripts | Faces, voices and names in an unapproved tool | Approved tools only, consent covers AI, anonymise first |
Most of these enter where nobody is looking: somewhere between the recording and the slide. The next three sections take the worst ones in turn.
3. Invented quotes
A quote is the strongest evidence in a report, and the easiest to fake by accident. There are two ways it happens:
- The transcript invents it. A study of OpenAI's Whisper speech-to-text found that roughly 1% of transcriptions contained whole phrases or sentences that "did not exist in any form in the underlying audio", and 38% of those hallucinations included harms such as violence or false associations. They happened more often for speakers with longer pauses, as is common with aphasia (Koenecke et al., ACM FAccT 2024). One percent sounds small, until it's the one quote on your slide.
- The summary invents it. Ask a model for "key quotes" and it can return a cleaner, more quotable version of what someone said, or merge two people into one. NN/g lists "made-up data or details that are not rooted in research" as a risk of AI in reporting (NN/g, 2024).
| Quote in the draft | Source | In the recording | Result |
|---|---|---|---|
| "I don't mind paying more. I mind not knowing why." | P4 · 23:40 | Word for word | Keep |
| "The app is useless for couples." | AI summary, "key quotes" | P2 said: "For us two it's a bit much, honestly. We mostly just use the chat." | Paraphrase made stronger: replace with the real words |
| "Nobody in our flat trusts the totals." | Transcript, P7 · 31:05 | Not found. At 31:05 there are 8 seconds of silence while P7 looks for a receipt | Invented by transcription: remove |
| "I'm always the one chasing people." | P1 · 12:18 and P6 · 40:02 | P1 said it; P6 said "I end up chasing everyone" | Two people merged: keep P1, quote P6 separately |
The rule this course recommends: no quote goes in a report until someone has heard it in the recording, with a participant code and a timestamp next to it. That's also what makes the repository trustworthy: the nugget's evidence is the recording, not the transcript.
4. Synthetic users, and synthetic respondents in your data
Two different problems share a name.
Synthetic users on purpose. Tools offer AI personas that answer interview questions. NN/g re-ran three of its own studies with them: a real learner admitted finishing three of seven courses, the synthetic one claimed all of them; real participants avoided course forums, synthetic ones praised them (Rosala and Moran, NN/g, 2024). In 2026 NN/g compared 604 people with 120 synthetic respondents on one open question: the same headline, but people used more than 50 different labels and the synthetic respondents about a dozen. Its verdict: "Research with synthetic users is still not user research, because no actual users are being studied" (Rachel Banawa, NN/g, 2026). Use them, at most, to pilot a guide or generate hypotheses, and label any output "not user research".
Synthetic respondents you didn't ask for. This one is newer and worse. A 2025 paper in PNAS built an AI agent that answered online surveys with a consistent persona, passed 99.8% of 6,000 standard attention-check trials, and could infer the researcher's hypothesis and produce data that confirmed it (Westwood, PNAS, 2025). NN/g's practical advice for surveys: no single signal proves a bot, so combine several, such as completion times far below your pilot's median, long generic open answers of suspiciously similar length, and batches arriving together, and document how many responses you removed and why (Rachel Banawa, NN/g, 2026). The same caution applies to unmoderated tests and screeners with incentives: watch some sessions, and verify people before paying them.
5. The team side: policy, consent and disclosure
- Make the safe path the easy one. Laura Klein found researchers pasting transcripts and customer data into public tools because nobody had told them not to: "The best security policies are ones that make the secure option the easiest option" (NN/g, 2025). An approved tool, a one-page policy and a shared prompt library do more than a 50-page document.
- Say it in the consent form. If recordings will be transcribed or analysed by an AI service, participants should know before they agree (see Research ethics and consent and Research data and privacy).
- Disclose in the report. One line in the method: which steps used AI, which tool, and what a person checked. People who make decisions on your evidence deserve to know how it was made.
- Keep watching the sessions. AI-moderated interviews can help with structured feedback, but in NN/g's study only 3 of 10 research leaders felt the conversation was natural, and the tools couldn't adapt questions (Maria Rosala, NN/g, 2026). And for usability tests, NN/g's advice is direct: "Don't attempt to use AI analysis tools for usability testing", because a transcript shows what people said, not what they did (NN/g, 2024).
6. Common mistakes
- Quoting a transcript without listening to the recording.
- Asking AI for "the key quotes" and pasting them into the deck.
- Presenting synthetic-user output in the same slides as real findings.
- Cleaning survey data silently, or not at all.
- Analysing a usability test from its transcript.
- Uploading recordings to a tool participants weren't told about.
7. Vocabulary
| Term | Meaning |
|---|---|
| Hallucination | Output an AI presents as fact that isn't supported by its input |
| Synthetic user | An AI-generated persona that answers as if it were a participant |
| Survey bot | Automated or AI-assisted responses submitted to a survey as if by a person |
| Mode collapse | When AI answers bunch around a few typical options and lose the rare ones |
| Attention check | A survey question designed to catch careless or automated respondents |
| Disclosure | Saying in the report where and how AI was used |
8. Try it (no tools needed)
- Take a report or deck with participant quotes in it.
- Pick three quotes and find each one in the recording, not the transcript.
- Mark each: word for word, paraphrased, or not found.
Reflection questions: How many steps between the recording and the slide did each quote go through? Who on your team would notice an invented sentence?
9. If you already practise
- Add "quotes checked against recordings" to your report template's method section.
- Write your team's AI rules for research on one page: approved tools, what data may go in, what must be checked by a person, how to disclose.
- Pilot every survey and record the median time, so you have a baseline for spotting bots.
Checklist
Learn more
- Kate Moran and Maria Rosala, Accelerating Research with AI (NN/g, 2024)
- Maria Rosala and Kate Moran, Synthetic Users: If, When, and How to Use AI-Generated "Research" (NN/g, 2024) · Rachel Banawa, Synthetic Users vs. Human Respondents (NN/g, 2026)
- Rachel Banawa, Kick the Bots Out of Your Survey Data (NN/g, 2026)
- Laura Klein, Stop Making Your Team Figure Out AI on Their Own (NN/g, 2025)
- Koenecke et al., Careless Whisper: Speech-to-Text Hallucination Harms (ACM FAccT, 2024)
- Sean Westwood, The potential existential threat of large language models to online survey research (PNAS, 2025)
Books, courses and videos to go deeper
- Just Enough ResearchErika HallBook · Mule Books, 2024 edition (Amazon)
- Accelerating Research with AINN/gCourse · paid, live online course (external site)
Book links go to Amazon. As an Amazon Associate I earn from qualifying purchases, at no extra cost to you.
Related
AI in UX research · Synthetic users and digital twins · AI-assisted synthesis, step by step · Notes, recordings and transcripts · Bias in surveys