Marcus Andrey Study
Ler em português Library

AI for UX · Step 5 · 5 min read

Evals for designers

Evals are the AI version of usability testing plus QA: structured tests of AI output against criteria. Why one good answer proves nothing, how designers write the test cases and rubrics, how automated judges work and fail, and a simple eval you can run without code.

"I tried it and it worked" is not an evaluation. AI gives different answers to the same question, and it can be excellent on the cases you tried and poor on the ones your users will bring. Evals are how teams find out how good an AI feature really is, before and after launch. Designers are often the best people to write them, because evals start with the question every designer asks: what does good look like for this user?

What you'll be able to do after reading:

  • Explain what an eval is, in plain words, and why one output proves nothing
  • Write test cases drawn from real users and real failures
  • Turn "good" into a rubric someone else can apply
  • Know when to trust an automated judge, and how to check it
  • Run a small eval with a spreadsheet and no code

1. What an eval is

An eval is a structured test of AI output: a set of inputs, a definition of what a good answer looks like, and a way to grade many answers consistently. Think of it as usability testing plus QA for something that changes its answer every time.

NN/g treats it like a quantitative study: "One AI output is an example, not an evaluation." Test with many representative inputs, repeat runs, and look at the spread, not one impressive answer (NN/g, Aug 2026).

A metal cylinder, a national standard kilogram, under two glass bell jars on a stand.
A national kilogram standard: every scale is checked against a reference everyone agrees on. An eval set is that reference for an AI feature. Photo: National Institute of Standards and Technology, Wikimedia Commons, public domain.

2. The eval loop

  1. 1Define good

    Specific, measurable criteria.

  2. 2Collect cases

    Real inputs and real failures, plus edge cases.

  3. 3Run

    Every case, more than once.

  4. 4Grade

    With code, a model or people, using a rubric.

  5. 5Read and improve

    Read outputs; fix prompt, data or design.

repeat whenever the model, prompt or data changes

The eval loop. Every real failure after launch becomes a new case. Diagram: Marcus Andrey, CC BY 4.0.

Test cases: real, varied, nasty

  • Start from reality. Anthropic's guidance for agents: "20-50 simple tasks drawn from real failures is a great start" (Anthropic). Support tickets, research notes and bug reports are full of them.
  • Cover the edges: long inputs, typos, other languages, ambiguous requests, inputs that should be refused.
  • Keep a held-out set you never tune prompts on, so you know whether you're improving or just fitting.

Success criteria: specific and measurable

Anthropic recommends criteria that are specific and measurable (Anthropic docs). "Friendly" isn't; "states the next step in one sentence, under 140 characters, without blaming the user" is.


3. Rubrics: turning "good" into something gradable

A rubric is a short list of criteria, each scored the same way by anyone:

Criterion Pass Fail
Correct Every fact matches the source Any invented fact, number or feature
Complete Answers the whole request Misses part of it
On-voice Matches the product's tone and terms Generic or off-brand
Actionable Ends with a clear next step Leaves the person stuck
Safe Refuses what it should, protects personal data Leaks or overreaches

Pass/fail per criterion is easier to apply consistently than 1-to-10 scores. OpenAI's guidance warns against "vibe-based evals": judging by feel instead of by written criteria (OpenAI).


4. Who grades: code, models or people

CodeRules in code

Length, format, required words, links that work.

Fast and exact, but only for what can be checked mechanically
ModelAn AI judge

Another model grades with a rubric.

Scales well; biased to first and longer answers; calibrate it
PeopleHuman graders

Designers, writers, experts with the rubric.

The reference; slow; needed to check the others
Three kinds of grader. Most good evals combine them, and check the automated ones against people. Diagram: Marcus Andrey, CC BY 4.0.

Automated judges ("LLM-as-a-judge") use one AI model to grade another's answers. A well-known study found strong judges agree with humans over 80% of the time, about as often as humans agree with each other, but also documented biases: favouring the first answer shown, favouring longer answers, and favouring their own style (Zheng et al., NeurIPS 2023). Both Anthropic and OpenAI say model graders must be calibrated against human graders before you rely on them.


5. Where designers make the difference

  • Defining good: criteria come from user needs, content guidelines and accessibility, which designers own.
  • Writing realistic cases: from research and support, including the people most likely to be failed. See Ethics, bias and inclusion in AI design.
  • Reading the outputs: Anthropic's advice is simply to read the transcripts. Patterns of failure are design insights.
  • Keeping the eval alive: every new real failure becomes a new test case.

6. Vocabulary

Term Meaning
Evals Structured tests of AI output against criteria
Test set The inputs an eval uses
Held-out set Cases kept aside, never used to tune prompts
Rubric Written criteria for grading an answer
LLM-as-a-judge Using an AI model to grade AI answers
Regression Something that used to work and now doesn't

7. Try it (no code needed)

  1. Pick one AI feature (a summary, a reply suggestion, a rewritten error message).
  2. In a spreadsheet, write 20 real inputs, including five difficult ones.
  3. Write a five-criterion pass/fail rubric (section 3).
  4. Run every input through the feature, paste the outputs, and grade them. Count passes per criterion.
  5. Run the same 20 again a day later. What changed?

Reflection questions: Which criterion failed most? Was it a prompt problem, a data problem or a design problem? Who in your team should own the test set?


8. If you lead a team

  • Make an eval set a requirement before any AI feature ships, with design writing the criteria.
  • Re-run the eval whenever the model, prompt or data changes.
  • Keep the eval set in a shared place, and add every real failure to it.

Checklist

Learn more

Books, courses and videos to go deeper

Designing for uncertainty · Measuring AI features · AI design review process · Evals · Hallucination · Ethics, bias and inclusion in AI design

Saved only on this device.

↑ ↓ to move · Enter to open · Esc to close