Marcus Andrey Study
Ler em português Library

AI for UX · Step 5 · 5 min read

Measuring AI features

How to know whether an AI feature actually helps: why people's sense of speed can be wrong, what the best studies found, which metrics to track (acceptance, correction, task success, errors introduced, retention), the limits of thumbs up and down, and a measurement plan you can adapt.

"It feels faster" is the most common and least reliable claim about AI. In one careful study, experienced developers using AI were measurably slower, while believing they were faster. Evals tell you how good the AI's answers are; measurement tells you whether people are better off with the feature in their real work. You need both.

What you'll be able to do after reading:

  • Explain why perceived productivity and measured productivity can disagree
  • Summarise what the best-known studies of AI at work found
  • Choose metrics for an AI feature, beyond usage
  • Know the limits of thumbs up and down
  • Write a simple measurement plan

1. Perceived is not measured

  1. Copilot, one coding task: faster55.8%
  2. Support agents: more issues solved per hour15%
  3. METR: developers expected to be faster24%
  4. METR: developers felt faster afterwards20%
  5. METR: measured, they were slower19%
Three well-known studies (Peng et al. 2023; Brynjolfsson et al. 2025; METR 2025). The effect depends on task, person and setting, and people's sense of it can be wrong. Diagram: Marcus Andrey, CC BY 4.0.
Study What it found
Peng et al., 2023, controlled experiment Developers with GitHub Copilot finished one programming task 55.8% faster (paper)
Brynjolfsson, Li & Raymond, 2025, Quarterly Journal of Economics Customer-support agents with an AI assistant resolved 15% more issues per hour; the biggest gains were for less experienced agents, and the most skilled saw small declines in quality (NBER)
METR, July 2025, controlled experiment 16 experienced open-source developers were 19% slower with AI on their own projects, but expected to be 24% faster and still believed they had been 20% faster (paper)
METR, February 2026 update Follow-up data with returning developers suggested a speed-up instead, but METR warns it is unreliable because of selection bias (METR)

The lesson isn't "AI is fast" or "AI is slow". It's that the answer depends on the task, the person and the setting, and people's own sense of it can be wrong. Measure.

A mechanical stopwatch from around 1970, standing on a rock in the snow.
How long something felt and how long it took are different numbers. Only one of them comes from a stopwatch. Photo: R. Henrik Nilsson, Wikimedia Commons, CC BY 4.0.

2. What to measure

1Task success

Did people finish what they came to do?

2Time

Measured time on task, not felt speed.

3Acceptance

Share of suggestions used.

4Correction

Share edited, undone or discarded.

5Errors introduced

Mistakes that would not have happened without AI.

6Trust over time

Does checking drop while errors stay low?

Six measures for an AI feature. Usage alone says it is used, not that it helps. Diagram: Marcus Andrey, CC BY 4.0.

Google's HEART framework (Happiness, Engagement, Adoption, Retention, Task success) still works as a frame for AI features (Rodden et al., 2010); this course's suggestion is to add two AI-specific measures to it: correction (how often people fix the AI) and errors introduced (mistakes that would not have happened without it).

Acceptance rate: useful, and easy to misread

For code suggestions, the share of suggestions people accept was the best predictor of how productive developers felt (Ziegler et al., 2022). Accepted is not the same as correct: pair acceptance with correction and errors.


3. The limits of thumbs up and down

  • It measures pleasure, not truth. OpenAI reported that training on thumbs-up and thumbs-down data helped make a model too agreeable: user feedback "can sometimes favor more agreeable responses" (OpenAI).
  • Few people click. Explicit ratings are rare and come from the very happy or very annoyed.
  • Use implicit signals too. PAIR separates explicit feedback (ratings) from implicit feedback (accepting, editing, undoing, abandoning), and asks that both the system and the user understand what the feedback means (PAIR).

4. A measurement plan

Question Example for an "AI reply suggestion" feature
Goal Agents answer customers faster without more errors
Task success Share of conversations resolved without escalation
Time Median time to first reply, with and without the feature
Acceptance and correction Suggestions sent as-is / edited / discarded
Errors introduced Wrong facts sent to customers, from audits of a sample
Trust over time Does editing drop over weeks (good) while errors rise (bad: over-trust)?
Comparison A group without the feature, or before/after with care
Qualitative Five interviews a month: when do people ignore the AI, and why?

5. Vocabulary

Term Meaning
Acceptance rate Share of AI suggestions people use
Correction rate Share of AI suggestions people edit or undo
Errors introduced Mistakes caused by AI that wouldn't otherwise happen
HEART Happiness, Engagement, Adoption, Retention, Task success
Controlled experiment Comparing groups with and without the feature, assigned fairly
Explicit / implicit feedback Ratings people give, vs signals from what they do

6. Try it (no code needed)

  1. Pick one AI feature you use or design.
  2. Write its goal in one sentence.
  3. Choose one metric for each of: task success, time, correction, errors introduced, trust over time.
  4. For each, write how you'd collect it and what would make you turn the feature off.

Reflection questions: Which of your AI features is judged only by usage? What would you need to see to believe it actually helps?


7. If you lead a team

  • Don't report AI success with usage alone. Pair it with task success and errors introduced.
  • Plan a comparison (a holdout group or a careful before/after) before launch, not after.
  • Share results honestly, including features that didn't help.

Checklist

Learn more

Books, courses and videos to go deeper

Evals for designers · Designing for uncertainty · Agentic design · Feedback loops · Automation bias · Building an AI agent roadmap for your UX team

Saved only on this device.

↑ ↓ to move · Enter to open · Esc to close