Marcus Andrey Study
Ler em português Library

UX Research · Step 7 · 7 min read

AI in UX research, carefully

AI can save hours in research and quietly put things in your report that nobody said. The risks that matter most for a research team, from invented quotes and synthetic users to AI respondents inside your own survey data, and the checks that keep the evidence honest.

The danger isn't that AI gets research wrong. It's that it gets it wrong fluently. A transcript with one invented sentence looks exactly like a correct one. A synthetic user sounds like a customer. A bot's survey answer reads like a thoughtful person's. Research is the part of the product process whose whole job is to be true, so the checks have to be stronger here than anywhere else.

What you'll be able to do after reading:

  • Name the six risks AI brings into research, and where each one enters
  • Check quotes against the recording before they reach a report
  • Explain to a stakeholder why synthetic users are not user research
  • Spot AI respondents in a survey, and say what you removed and why

1. Start with the AI for UX lessons

This lesson assumes you know where AI helps across the research cycle. The AI for UX course covers that: AI in UX research goes stage by stage, Synthetic users and digital twins reviews the studies on AI respondents, and AI-assisted synthesis, step by step gives a method for analysis with AI that keeps the rigour. The short version, from NN/g: AI might "miss, misinterpret, or even manufacture insights" (Kate Moran and Maria Rosala, NN/g, 2024).

Here we go one step further, into the risks a research team has to own: the ones that put false evidence in front of people who will make decisions with it.


2. Six risks, and the check for each

RiskWhere it entersWhat it looks likeThe check
Invented quotesTranscription, summariesA clean, quotable sentence nobody saidHear every quote in the recording; add code and timestamp
Invented or missed findingsAI analysisFrequent themes in, the important rare one outRead sessions first; trace each insight to raw data
Synthetic users as evidencePlanning, "validation"Agreeable personas, a dozen answers where people give fiftyLabel it "not user research"; never in the findings
Bots in your dataSurveys, unmoderated testsFluent answers, fast completions, batchesSeveral signals together; document what you removed
Lost behaviourUsability analysis from transcriptsWhat people said, not what they didWatch the sessions
Participants' dataUploading recordings and transcriptsFaces, voices and names in an unapproved toolApproved tools only, consent covers AI, anonymise first
Six risks AI brings into research, each with the point where it gets in and the check that stops it. Most checks are the same habit: go back to the raw evidence.Table: Marcus Andrey, CC BY 4.0, from NN/g (2024, 2025, 2026), Koenecke et al. (2024) and Westwood (2025), linked in the lesson.

Most of these enter where nobody is looking: somewhere between the recording and the slide. The next three sections take the worst ones in turn.


3. Invented quotes

A quote is the strongest evidence in a report, and the easiest to fake by accident. There are two ways it happens:

  • The transcript invents it. A study of OpenAI's Whisper speech-to-text found that roughly 1% of transcriptions contained whole phrases or sentences that "did not exist in any form in the underlying audio", and 38% of those hallucinations included harms such as violence or false associations. They happened more often for speakers with longer pauses, as is common with aphasia (Koenecke et al., ACM FAccT 2024). One percent sounds small, until it's the one quote on your slide.
  • The summary invents it. Ask a model for "key quotes" and it can return a cleaner, more quotable version of what someone said, or merge two people into one. NN/g lists "made-up data or details that are not rooted in research" as a risk of AI in reporting (NN/g, 2024).
Quote in the draftSourceIn the recordingResult
"I don't mind paying more. I mind not knowing why."P4 · 23:40Word for wordKeep
"The app is useless for couples."AI summary, "key quotes"P2 said: "For us two it's a bit much, honestly. We mostly just use the chat."Paraphrase made stronger: replace with the real words
"Nobody in our flat trusts the totals."Transcript, P7 · 31:05Not found. At 31:05 there are 8 seconds of silence while P7 looks for a receiptInvented by transcription: remove
"I'm always the one chasing people."P1 · 12:18 and P6 · 40:02P1 said it; P6 said "I end up chasing everyone"Two people merged: keep P1, quote P6 separately
A filled-in quote check for a draft report. Four quotes, one kept as it was. The silent stretch is where speech-to-text is most likely to invent words, which is why the recording, not the transcript, is the evidence.Example study, invented for this course. Table: Marcus Andrey, CC BY 4.0. Hallucinations during pauses: Koenecke et al., Careless Whisper (ACM FAccT, 2024).

The rule this course recommends: no quote goes in a report until someone has heard it in the recording, with a participant code and a timestamp next to it. That's also what makes the repository trustworthy: the nugget's evidence is the recording, not the transcript.


4. Synthetic users, and synthetic respondents in your data

Two different problems share a name.

Synthetic users on purpose. Tools offer AI personas that answer interview questions. NN/g re-ran three of its own studies with them: a real learner admitted finishing three of seven courses, the synthetic one claimed all of them; real participants avoided course forums, synthetic ones praised them (Rosala and Moran, NN/g, 2024). In 2026 NN/g compared 604 people with 120 synthetic respondents on one open question: the same headline, but people used more than 50 different labels and the synthetic respondents about a dozen. Its verdict: "Research with synthetic users is still not user research, because no actual users are being studied" (Rachel Banawa, NN/g, 2026). Use them, at most, to pilot a guide or generate hypotheses, and label any output "not user research".

Synthetic respondents you didn't ask for. This one is newer and worse. A 2025 paper in PNAS built an AI agent that answered online surveys with a consistent persona, passed 99.8% of 6,000 standard attention-check trials, and could infer the researcher's hypothesis and produce data that confirmed it (Westwood, PNAS, 2025). NN/g's practical advice for surveys: no single signal proves a bot, so combine several, such as completion times far below your pilot's median, long generic open answers of suspiciously similar length, and batches arriving together, and document how many responses you removed and why (Rachel Banawa, NN/g, 2026). The same caution applies to unmoderated tests and screeners with incentives: watch some sessions, and verify people before paying them.


  • Make the safe path the easy one. Laura Klein found researchers pasting transcripts and customer data into public tools because nobody had told them not to: "The best security policies are ones that make the secure option the easiest option" (NN/g, 2025). An approved tool, a one-page policy and a shared prompt library do more than a 50-page document.
  • Say it in the consent form. If recordings will be transcribed or analysed by an AI service, participants should know before they agree (see Research ethics and consent and Research data and privacy).
  • Disclose in the report. One line in the method: which steps used AI, which tool, and what a person checked. People who make decisions on your evidence deserve to know how it was made.
  • Keep watching the sessions. AI-moderated interviews can help with structured feedback, but in NN/g's study only 3 of 10 research leaders felt the conversation was natural, and the tools couldn't adapt questions (Maria Rosala, NN/g, 2026). And for usability tests, NN/g's advice is direct: "Don't attempt to use AI analysis tools for usability testing", because a transcript shows what people said, not what they did (NN/g, 2024).

6. Common mistakes

  • Quoting a transcript without listening to the recording.
  • Asking AI for "the key quotes" and pasting them into the deck.
  • Presenting synthetic-user output in the same slides as real findings.
  • Cleaning survey data silently, or not at all.
  • Analysing a usability test from its transcript.
  • Uploading recordings to a tool participants weren't told about.

7. Vocabulary

Term Meaning
Hallucination Output an AI presents as fact that isn't supported by its input
Synthetic user An AI-generated persona that answers as if it were a participant
Survey bot Automated or AI-assisted responses submitted to a survey as if by a person
Mode collapse When AI answers bunch around a few typical options and lose the rare ones
Attention check A survey question designed to catch careless or automated respondents
Disclosure Saying in the report where and how AI was used

8. Try it (no tools needed)

  1. Take a report or deck with participant quotes in it.
  2. Pick three quotes and find each one in the recording, not the transcript.
  3. Mark each: word for word, paraphrased, or not found.

Reflection questions: How many steps between the recording and the slide did each quote go through? Who on your team would notice an invented sentence?


9. If you already practise

  • Add "quotes checked against recordings" to your report template's method section.
  • Write your team's AI rules for research on one page: approved tools, what data may go in, what must be checked by a person, how to disclose.
  • Pilot every survey and record the median time, so you have a baseline for spotting bots.

Checklist

Learn more

Books, courses and videos to go deeper

Book links go to Amazon. As an Amazon Associate I earn from qualifying purchases, at no extra cost to you.

AI in UX research · Synthetic users and digital twins · AI-assisted synthesis, step by step · Notes, recordings and transcripts · Bias in surveys

Saved only on this device.

↑ ↓ to move · Enter to open · Esc to close