AI for UX · Step 5 · 5 min read
Measuring AI features
How to know whether an AI feature actually helps: why people's sense of speed can be wrong, what the best studies found, which metrics to track (acceptance, correction, task success, errors introduced, retention), the limits of thumbs up and down, and a measurement plan you can adapt.
"It feels faster" is the most common and least reliable claim about AI. In one careful study, experienced developers using AI were measurably slower, while believing they were faster. Evals tell you how good the AI's answers are; measurement tells you whether people are better off with the feature in their real work. You need both.
What you'll be able to do after reading:
- Explain why perceived productivity and measured productivity can disagree
- Summarise what the best-known studies of AI at work found
- Choose metrics for an AI feature, beyond usage
- Know the limits of thumbs up and down
- Write a simple measurement plan
1. Perceived is not measured
- Copilot, one coding task: faster55.8%
- Support agents: more issues solved per hour15%
- METR: developers expected to be faster24%
- METR: developers felt faster afterwards20%
- METR: measured, they were slower19%
| Study | What it found |
|---|---|
| Peng et al., 2023, controlled experiment | Developers with GitHub Copilot finished one programming task 55.8% faster (paper) |
| Brynjolfsson, Li & Raymond, 2025, Quarterly Journal of Economics | Customer-support agents with an AI assistant resolved 15% more issues per hour; the biggest gains were for less experienced agents, and the most skilled saw small declines in quality (NBER) |
| METR, July 2025, controlled experiment | 16 experienced open-source developers were 19% slower with AI on their own projects, but expected to be 24% faster and still believed they had been 20% faster (paper) |
| METR, February 2026 update | Follow-up data with returning developers suggested a speed-up instead, but METR warns it is unreliable because of selection bias (METR) |
The lesson isn't "AI is fast" or "AI is slow". It's that the answer depends on the task, the person and the setting, and people's own sense of it can be wrong. Measure.
2. What to measure
Did people finish what they came to do?
Measured time on task, not felt speed.
Share of suggestions used.
Share edited, undone or discarded.
Mistakes that would not have happened without AI.
Does checking drop while errors stay low?
Google's HEART framework (Happiness, Engagement, Adoption, Retention, Task success) still works as a frame for AI features (Rodden et al., 2010); this course's suggestion is to add two AI-specific measures to it: correction (how often people fix the AI) and errors introduced (mistakes that would not have happened without it).
Acceptance rate: useful, and easy to misread
For code suggestions, the share of suggestions people accept was the best predictor of how productive developers felt (Ziegler et al., 2022). Accepted is not the same as correct: pair acceptance with correction and errors.
3. The limits of thumbs up and down
- It measures pleasure, not truth. OpenAI reported that training on thumbs-up and thumbs-down data helped make a model too agreeable: user feedback "can sometimes favor more agreeable responses" (OpenAI).
- Few people click. Explicit ratings are rare and come from the very happy or very annoyed.
- Use implicit signals too. PAIR separates explicit feedback (ratings) from implicit feedback (accepting, editing, undoing, abandoning), and asks that both the system and the user understand what the feedback means (PAIR).
4. A measurement plan
| Question | Example for an "AI reply suggestion" feature |
|---|---|
| Goal | Agents answer customers faster without more errors |
| Task success | Share of conversations resolved without escalation |
| Time | Median time to first reply, with and without the feature |
| Acceptance and correction | Suggestions sent as-is / edited / discarded |
| Errors introduced | Wrong facts sent to customers, from audits of a sample |
| Trust over time | Does editing drop over weeks (good) while errors rise (bad: over-trust)? |
| Comparison | A group without the feature, or before/after with care |
| Qualitative | Five interviews a month: when do people ignore the AI, and why? |
5. Vocabulary
| Term | Meaning |
|---|---|
| Acceptance rate | Share of AI suggestions people use |
| Correction rate | Share of AI suggestions people edit or undo |
| Errors introduced | Mistakes caused by AI that wouldn't otherwise happen |
| HEART | Happiness, Engagement, Adoption, Retention, Task success |
| Controlled experiment | Comparing groups with and without the feature, assigned fairly |
| Explicit / implicit feedback | Ratings people give, vs signals from what they do |
6. Try it (no code needed)
- Pick one AI feature you use or design.
- Write its goal in one sentence.
- Choose one metric for each of: task success, time, correction, errors introduced, trust over time.
- For each, write how you'd collect it and what would make you turn the feature off.
Reflection questions: Which of your AI features is judged only by usage? What would you need to see to believe it actually helps?
7. If you lead a team
- Don't report AI success with usage alone. Pair it with task success and errors introduced.
- Plan a comparison (a holdout group or a careful before/after) before launch, not after.
- Share results honestly, including features that didn't help.
Checklist
Learn more
- Studies: Peng et al., The Impact of AI on Developer Productivity (2023) · Brynjolfsson, Li & Raymond, Generative AI at Work · METR, experienced developers study (2025) · METR update (Feb 2026) · Ziegler et al., acceptance and perceived productivity (2022)
- Frameworks: Google HEART (Rodden et al., 2010) · PAIR: Feedback + Control
Books, courses and videos to go deeper
Related
Evals for designers · Designing for uncertainty · Agentic design · Feedback loops · Automation bias · Building an AI agent roadmap for your UX team