Marcus Andrey Study
Ler em português Library

AI for UX · Labs · 4 min read

Write an eval set

A hands-on lab: build a small evaluation for an AI feature in a spreadsheet. Collect real cases, read the outputs, name the failures, count them, turn them into pass/fail criteria, and re-run after every change. Optional: automate it with promptfoo.

The most useful eval work isn't a tool. It's reading. People who build AI products for a living say it plainly: "Error analysis is the most important activity in evals" (Husain and Shankar, Sep 2026). You read real outputs, write down what went wrong, group the problems, and count them. That's qualitative analysis, the same skill as synthesising usability findings. This lab puts it in a spreadsheet.

What you'll be able to do after this lab:

  • Collect a test set from real inputs
  • Read outputs and name failures, then group them into categories
  • Turn the biggest categories into pass/fail criteria
  • Re-run the set after every change and compare

For the ideas behind this lab, see Evals for designers.


1. The loop

  1. 1Collect

    30 or more real cases.

  2. 2Read

    Every output; a note on each.

  3. 3Group

    Notes into a few categories.

  4. 4Count

    Failures per category; start with the biggest.

  5. 5Criteria

    Pass/fail rules anyone can apply.

After every change: run the same cases again.

Error analysis: read, name, group, count, then write the criteria. Diagram: Marcus Andrey, CC BY 4.0.
Tally marks in blue ink on squared paper, grouped in fives, counting visitors by day.
Error analysis ends in something this simple: a mark for every failure, in the right column. The biggest column is where you start. Photo: Gunnar Creutz, Wikimedia Commons, public domain.

2. Set up the spreadsheet

One row per case, these columns:

Input Output Notes (what's wrong) Category Pass?
A real request or error What the AI produced In your own words Filled in later yes / no

Where cases come from: support tickets, research notes, the AI feature's own logs (with permission), and the skill you built in Build your first skill. Include the difficult ones: typos, long inputs, other languages, requests that should be refused.

Husain and Shankar suggest you start by annotating at least 30 cases yourself, and aim to review at least 100 over time.

Download: eval set template (CSV), with these columns and three example rows. Open it in any spreadsheet app.


3. Read and write notes (open coding)

Read every output. In Notes, write what's wrong in plain words, as specifically as you can: "invented a refund policy", "action label is a noun", "ignores the second question". Don't decide the categories yet. There is, as they put it, "no replacement for looking at real interactions."

4. Group and count (axial coding)

When you have about 30 notes, group similar ones into a few categories (for example invented facts, wrong tone, missing action). Fill in the Category column and count the failures per category. The biggest category is where to work first.

5. Turn categories into pass/fail criteria

For each big category, write one criterion that anyone could apply: "Every fact matches the source: pass or fail." Use pass/fail, not 1 to 5: binary labels "force clearer thinking and more consistent labeling". Now the Pass? column has rules.

6. Change one thing, re-run, compare

Change the prompt, the skill, the model or the design. Run the same cases again into new columns and compare the pass rate per criterion. Keep every new real failure as a new row.


7. Optional: automate it with promptfoo

When the spreadsheet gets long, promptfoo can run it for you. It's open source (MIT), joined OpenAI in March 2026 and stays MIT, reads test cases from a CSV file or Google Sheets, and works with local models such as Ollama (docs). It needs Node.js 22.22 or later.

npx promptfoo@latest init     # creates a starter config
npx promptfoo@latest eval     # runs every case
npx promptfoo@latest view     # opens the results in the browser

8. Common mistakes

  • Starting with a score before reading any outputs.
  • Inventing test cases instead of using real ones.
  • 1-to-5 scales nobody applies the same way.
  • Trusting an automated judge you never compared with your own grading. See Evals for designers.
  • Not re-running after a model or prompt change.

9. Vocabulary

Term Meaning
Evals Structured tests of AI output against criteria
Test set The cases an eval uses
Error analysis Reading outputs, naming failures and counting them
Open coding Writing free notes on each output, without categories yet
Axial coding Grouping the notes into categories
Pass rate The share of cases that meet a criterion

10. Try it

  1. Pick one AI feature or skill.
  2. Collect 30 real inputs in a spreadsheet and run them through it.
  3. Write a note on every output, then group and count.
  4. Write pass/fail criteria for the two biggest categories.
  5. Change one thing, re-run, and compare.

Reflection questions: Which failure category surprised you? Which would a user notice first? Who in your team should own this spreadsheet?

Checklist

Learn more

Books, courses and videos to go deeper

Evals for designers · Measuring AI features · Build your first skill · AI-assisted synthesis, step by step · Evals · Hallucination

Saved only on this device.

↑ ↓ to move · Enter to open · Esc to close