AI for UX · Step 5 · 5 min read
Evals for designers
Evals are the AI version of usability testing plus QA: structured tests of AI output against criteria. Why one good answer proves nothing, how designers write the test cases and rubrics, how automated judges work and fail, and a simple eval you can run without code.
"I tried it and it worked" is not an evaluation. AI gives different answers to the same question, and it can be excellent on the cases you tried and poor on the ones your users will bring. Evals are how teams find out how good an AI feature really is, before and after launch. Designers are often the best people to write them, because evals start with the question every designer asks: what does good look like for this user?
What you'll be able to do after reading:
- Explain what an eval is, in plain words, and why one output proves nothing
- Write test cases drawn from real users and real failures
- Turn "good" into a rubric someone else can apply
- Know when to trust an automated judge, and how to check it
- Run a small eval with a spreadsheet and no code
1. What an eval is
An eval is a structured test of AI output: a set of inputs, a definition of what a good answer looks like, and a way to grade many answers consistently. Think of it as usability testing plus QA for something that changes its answer every time.
NN/g treats it like a quantitative study: "One AI output is an example, not an evaluation." Test with many representative inputs, repeat runs, and look at the spread, not one impressive answer (NN/g, Aug 2026).
2. The eval loop
- 1Define good
Specific, measurable criteria.
- 2Collect cases
Real inputs and real failures, plus edge cases.
- 3Run
Every case, more than once.
- 4Grade
With code, a model or people, using a rubric.
- 5Read and improve
Read outputs; fix prompt, data or design.
repeat whenever the model, prompt or data changes
Test cases: real, varied, nasty
- Start from reality. Anthropic's guidance for agents: "20-50 simple tasks drawn from real failures is a great start" (Anthropic). Support tickets, research notes and bug reports are full of them.
- Cover the edges: long inputs, typos, other languages, ambiguous requests, inputs that should be refused.
- Keep a held-out set you never tune prompts on, so you know whether you're improving or just fitting.
Success criteria: specific and measurable
Anthropic recommends criteria that are specific and measurable (Anthropic docs). "Friendly" isn't; "states the next step in one sentence, under 140 characters, without blaming the user" is.
3. Rubrics: turning "good" into something gradable
A rubric is a short list of criteria, each scored the same way by anyone:
| Criterion | Pass | Fail |
|---|---|---|
| Correct | Every fact matches the source | Any invented fact, number or feature |
| Complete | Answers the whole request | Misses part of it |
| On-voice | Matches the product's tone and terms | Generic or off-brand |
| Actionable | Ends with a clear next step | Leaves the person stuck |
| Safe | Refuses what it should, protects personal data | Leaks or overreaches |
Pass/fail per criterion is easier to apply consistently than 1-to-10 scores. OpenAI's guidance warns against "vibe-based evals": judging by feel instead of by written criteria (OpenAI).
4. Who grades: code, models or people
Length, format, required words, links that work.
Fast and exact, but only for what can be checked mechanicallyAnother model grades with a rubric.
Scales well; biased to first and longer answers; calibrate itDesigners, writers, experts with the rubric.
The reference; slow; needed to check the othersAutomated judges ("LLM-as-a-judge") use one AI model to grade another's answers. A well-known study found strong judges agree with humans over 80% of the time, about as often as humans agree with each other, but also documented biases: favouring the first answer shown, favouring longer answers, and favouring their own style (Zheng et al., NeurIPS 2023). Both Anthropic and OpenAI say model graders must be calibrated against human graders before you rely on them.
5. Where designers make the difference
- Defining good: criteria come from user needs, content guidelines and accessibility, which designers own.
- Writing realistic cases: from research and support, including the people most likely to be failed. See Ethics, bias and inclusion in AI design.
- Reading the outputs: Anthropic's advice is simply to read the transcripts. Patterns of failure are design insights.
- Keeping the eval alive: every new real failure becomes a new test case.
6. Vocabulary
| Term | Meaning |
|---|---|
| Evals | Structured tests of AI output against criteria |
| Test set | The inputs an eval uses |
| Held-out set | Cases kept aside, never used to tune prompts |
| Rubric | Written criteria for grading an answer |
| LLM-as-a-judge | Using an AI model to grade AI answers |
| Regression | Something that used to work and now doesn't |
7. Try it (no code needed)
- Pick one AI feature (a summary, a reply suggestion, a rewritten error message).
- In a spreadsheet, write 20 real inputs, including five difficult ones.
- Write a five-criterion pass/fail rubric (section 3).
- Run every input through the feature, paste the outputs, and grade them. Count passes per criterion.
- Run the same 20 again a day later. What changed?
Reflection questions: Which criterion failed most? Was it a prompt problem, a data problem or a design problem? Who in your team should own the test set?
8. If you lead a team
- Make an eval set a requirement before any AI feature ships, with design writing the criteria.
- Re-run the eval whenever the model, prompt or data changes.
- Keep the eval set in a shared place, and add every real failure to it.
Checklist
Learn more
- NN/g: One AI Output Is an Example, Not an Evaluation (Aug 2026)
- Vendors: Anthropic, Demystifying evals for AI agents (Jan 2026) · Anthropic, Define success criteria and build evaluations · OpenAI, Evaluation best practices
- Research: Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023)
Books, courses and videos to go deeper
Related
Designing for uncertainty · Measuring AI features · AI design review process · Evals · Hallucination · Ethics, bias and inclusion in AI design