Marcus Andrey Study
Ler em português Library

Framework · AI & design · 3 min read

Trust calibration

Designing AI features so users trust them exactly as much as they deserve: not blindly (automation bias) and not never (wasted value), and testing that they do.

Too littleUnder-trust

People ignore good suggestions and redo the work.

The feature is wasted
RightCalibrated trust

People rely on the AI where it is good and check where it is weak.

The goal
Too muchOver-trust

People stop checking and accept errors.

Automation bias
Trust calibration: the aim is trust that matches how reliable the AI really is. Diagram: Marcus Andrey, CC BY 4.0.

Use it when

  • A product team is designing an AI feature such as document extraction, anomaly detection or an agent that drafts actions.
  • Users ignore a useful AI feature and redo the work by hand.
  • Experts have stopped checking AI suggestions after a run of correct ones, and errors are slipping through.

How to use it

  1. Know how reliable the AI really is. Ask the team where it is strong and where it fails: which fields, which document types, which edge cases. You can only calibrate trust against real performance. Output: a short list of "reliable here, weak here".
  2. Show confidence and reasons. Show confidence as levels, not fake precise percentages, and say where the answer came from: "Extracted from invoice p.2, low confidence on HS code." Add source citations for answers based on documents or policies. Output: confidence and source patterns in the design.
  3. Expose failures honestly. Make the AI's role and limits clear. When it is unsure or stuck, say so and hand over with context instead of guessing. Make correction easy. Output: designed states for low confidence, failure and handover.
  4. Test with deliberately wrong suggestions. In usability tests, include realistic wrong suggestions and see whether users catch them. This is your over-trust check. Run a Task success and time on task comparison with and without the AI. Output: a catch rate for wrong suggestions, and task results.
  5. Measure trust over time. After launch, track acceptance rate, correction rate, time saved and errors introduced. A rising acceptance rate with steady errors can mean people stopped checking. Output: a small set of measures with a review date.

Worked example

An internal operations tool adds AI extraction of fields from scanned invoices. In the first test, operators accept almost every suggestion, including the wrong ones.

Signal What the team found Design change
Wrong suggestions in the test Operators accepted most of the planted wrong customs codes Low-confidence fields are highlighted and need an explicit confirm
Confidence display A "97%" score on every field was ignored Three levels (high, check, low) with the reason in a tooltip
Sources Operators could not see where a value came from Clicking a field shows the spot on the invoice
Under-trust One shift retyped every field "to be safe" High-confidence fields are pre-confirmed, with undo

In the second round, operators catch most planted errors and still save time on the fields that are reliable.

Watch out for

  • Fake precision. "97.3% confident" looks scientific but users cannot act on it. Use a few clear levels and say what to do at each one.
  • Testing only the happy path. If every suggestion in your test is correct, you learn nothing about over-trust. Plant realistic errors.
  • Hiding failures to look good. An AI that never admits uncertainty trains users to trust it blindly, until the first expensive mistake.
  • Measuring acceptance as success. A high acceptance rate can mean the AI is good, or that people stopped checking. Look at errors introduced too.

Template

Copy this into the feature spec for any AI feature.

AI feature:
Reliable for:                          Weak for:

Confidence shown as:    levels (e.g. high / check / low), not percentages
What the user does at each level:
  High:                 Check:                 Low:
Sources shown:          (where the answer came from)
Failure and handover:   (what the AI says when unsure or stuck)
Correction:             (how to fix it, and whether corrections feed back)

Over-trust test
  Planted wrong suggestions (how many, which kinds):
  Catch rate:
  Task result with AI / without AI:

Measures after launch:  acceptance · correction · time saved · errors introduced
Review on:

"See on Amazon" links are affiliate links. As an Amazon Associate I earn from qualifying purchases, at no extra cost to you.

Saved only on this device.

↑ ↓ to move · Enter to open · Esc to close