Marcus Andrey Study
Ler em português Library

AI for UX · Step 3 · 6 min read

Ethics, bias and inclusion in AI design

Where bias in AI comes from, real cases that affect design work, why people over-trust AI and how to design against it, the manipulative patterns the law now bans, and practical habits that keep AI-assisted work fair and inclusive.

AI learns from the world as it was written down, including its unfairness. It can repeat stereotypes in an image, understand some accents worse than others, or nudge people into decisions they wouldn't make. None of this needs bad intent. Designers are often the last people to see the output before users do, so knowing what to look for is part of the job.

What you'll be able to do after reading:

  • Explain the three sources of bias in AI, in plain words
  • Recognise bias in AI output you use for design work: images, copy, research summaries
  • Design against automation bias, so people don't trust AI more than it deserves
  • Know which manipulative design patterns are banned, and design honest AI features

1. Where bias comes from

The US National Institute of Standards and Technology (NIST) describes three kinds of bias in AI, and notes that they can appear "in the absence of… discriminatory intent" (NIST AI RMF):

1Systemic

Unfairness already in society and institutions, reflected in the data.

e.g. who held which jobs in old photos
2Statistical

Gaps and imbalances in the data and the method.

e.g. few recordings of some accents
3Human

How people read and act on the output.

e.g. trusting a confident answer
Three sources of bias in AI, as described by the NIST AI Risk Management Framework. Diagram: Marcus Andrey, CC BY 4.0.

2. Real cases designers should know

Case What happened Why it matters for design
Image generators and jobs Bloomberg analysed over 5,000 Stable Diffusion images (2023): images for every high-paying job were dominated by lighter skin, and more than 80% of images for "inmate" showed darker skin (Bloomberg) Generated personas, hero images and illustrations can carry stereotypes into your product
Speech recognition Five commercial systems made almost twice as many errors for Black speakers (word error rate 0.35) as for white speakers (0.19) (Koenecke et al., PNAS 2020) Voice features and AI transcription of research sessions work worse for some people
Friendly AI, more errors NN/g reports studies where "warm" chatbot personalities had 10–30% higher error rates, and warns against humanising AI: "design real tools, not fake friends" (NN/g, Jan 2026) Personality is a design choice with a cost

The Bloomberg piece is data journalism, not peer-reviewed research, but its method is public and the pattern has been repeated in other studies.


3. Automation bias: trusting the machine too much

Automation bias is the tendency to accept a system's suggestion even when it's wrong. It affects experts too:

Readers who were right about 80% of the time…

  1. Without misleading AI≈ 80%
  2. Very experienced, AI wrong45.5%
  3. Moderately experienced, AI wrong24.8%
  4. Inexperienced, AI wrong19.8%
Accuracy of 27 radiologists reading mammograms when a supposed AI suggested the wrong answer (Dratsch et al., Radiology, 2023). Experience helped, but did not protect. Diagram: Marcus Andrey, CC BY 4.0.

What this means for designing AI features:

  • Show how sure the AI is, and why, instead of presenting every answer with the same confidence. See Confidence indicators and Trust calibration.
  • Make checking easy: show the source, the data used, what changed.
  • Ask for a decision, not a click: "Review these 3 changes" beats one "Accept all" button.
  • Watch for decay: after a run of correct suggestions, people stop checking. Test with deliberately wrong suggestions to see whether users catch them.

4. Manipulation: what is now banned

Some design patterns are not only unethical but illegal in the EU:

  • The AI Act bans AI that manipulates or deceives people in ways that cause significant harm, or that exploits vulnerabilities such as age, disability or social or economic situation. These bans have applied since 2 February 2025 (Article 5).
  • The Digital Services Act bans dark patterns on online platforms: interfaces that "deceive or manipulate" users or impair free and informed decisions. It has applied since 17 February 2024 (DSA, Article 25).

In AI features, watch for:

  • An assistant that pushes the most profitable option while sounding neutral
  • Fake urgency or social proof generated on the fly ("3 people are looking at this")
  • A chatbot that pretends to be a person, or hides how to reach one
  • Personalisation that targets people when they are most vulnerable

5. Frameworks you can point to

You don't need to memorise them, but it helps to know they exist when you argue for a change:

Framework In one line
NIST AI Risk Management Framework (2023, with a Generative AI Profile from 2024) Four functions: govern, map, measure, manage; seven qualities of trustworthy AI, including "fair, with harmful bias managed"
UNESCO Recommendation on the Ethics of AI (2021, 193 member states) Values and principles including diversity and inclusiveness, fairness and human oversight
OECD AI Principles (2019, updated 2024) Five principles for trustworthy AI adopted by 47 countries

6. Practical habits

  • Specify people, don't leave it to defaults. In image prompts and generated personas, describe age, body, skin tone, disability and context on purpose, and vary them across a set. Review the set as a whole.
  • Check who is missing. In an AI research summary, ask: whose quotes were kept, whose were dropped? See AI in UX research.
  • Test with the people most likely to be failed: different accents, languages, abilities, devices, connection speeds.
  • Give users a way to say "this is wrong", and make sure someone reads it. See Feedback loops.
  • Name the trade-off. If a feature is faster for most people but worse for some, write it down and decide openly.

7. Vocabulary

Term Meaning
Algorithmic bias Systematic unfairness in an AI system's outputs
Automation bias Trusting a system's suggestion more than you should
Dark pattern An interface designed to trick people into choices they wouldn't otherwise make
Anthropomorphism Giving a system human traits (a name, feelings, a face)
Trust calibration Helping people trust a system exactly as much as it deserves

8. Try it (no code needed)

  1. Ask an approved image tool for "a photo of a doctor", four times. Then "a photo of a nurse", four times. Who appears? Who doesn't?
  2. Ask an assistant to write three personas for a banking app. Note the names, ages and situations it picks by default.
  3. Rewrite both prompts to specify a varied set of people, and compare.
  4. Look at one AI feature you use daily. Does it show how sure it is? How would you know if it were wrong?

Reflection questions: Which of your product's users is most likely to be failed by an AI feature? What would it take for your team to notice?


9. If you lead a team

  • Add "who could this fail?" to your design critique questions.
  • Include people at risk of being excluded in research and testing, not only "typical" users.
  • Keep a short list of red lines for AI features (no fake humans, no manufactured urgency, always a way to a person) and review new features against it.

Checklist

Learn more

Books, courses and videos to go deeper

AI design review process · Accessibility with and for AI · Algorithmic bias · Automation bias · Trust calibration · Agentic design · AI governance and data protection

Saved only on this device.

↑ ↓ to move · Enter to open · Esc to close