AI for UX · Step 3 · 7 min read
AI governance and data protection
What any designer should know in late 2026 about using AI with real data and designing AI features: data protection basics, data classes and approved tools, the EU AI Act timeline as amended in 2026, workers' rights, and governance you can build into the design itself.
Most AI risks in design work are data risks. A research recording pasted into the wrong tool, a customer list used to "test" a prompt, an AI feature that quietly scores employees. None of this needs bad intent; it needs a moment of not asking. This note gives you the questions to ask, and the rules behind them.
What you'll be able to do after reading:
- Apply the data protection basics to AI tools and research data
- Use data classes and an approved tools list to decide what may go where
- Read the EU AI Act timeline and know which parts concern designers
- Recognise when an AI feature touches workers' rights or high-risk uses
- Build transparency, oversight and logging into the AI features you design
1. Data protection: still the foundation
In the EU and many countries with similar laws, the General Data Protection Regulation (GDPR) applies to AI tools like any other processing of personal data:
| Principle | What it means in AI work | GDPR |
|---|---|---|
| A purpose and a lawful basis | Know why this data goes into this tool, and on what legal ground | Art. 5, 6 |
| Data minimisation | Use the least data needed; anonymise or pseudonymise before using AI | Art. 5 |
| A processor contract | The vendor processes data on your behalf under a written agreement (often called a DPA) | Art. 28 |
| An impact assessment for risky processing | New technology on personal data at scale may need a data protection impact assessment | Art. 35 |
| Limits on automated decisions | People have rights when decisions about them are made solely by automated means | Art. 22 |
Research recordings deserve special care: faces, voices and names are personal data. They become "special category" biometric data when processed to identify someone (GDPR).
"Training opt-out" is not a GDPR term but a contract setting: whether the vendor may use your content to improve its models. Check it for every tool.
2. Data classes and approved tools
Agree a simple set of data classes with your data protection officer and IT security, then map each approved tool to the classes it may receive:
Published guidelines, public documents.
Any approved toolSpecs, design-system docs, anonymised notes.
Approved business tools onlyCustomer names, transactions, pricing, unreleased plans.
Only tools approved for it; prefer anonymised dataRecordings, transcripts, employee data.
Specifically approved tools; anonymise first; involve the DPOThe approved tools list is the practical backbone. For each tool it says: which plan or contract (consumer and business plans have very different data terms), which data classes are allowed, whether training on your content is off, where data is processed, and who owns the tool internally. Personal accounts on consumer plans should never receive work data.
3. The EU AI Act, as amended in 2026
The AI Act (Regulation (EU) 2024/1689) was amended by the Digital Omnibus on AI (Regulation (EU) 2026/1744), in force since 27 July 2026, which moved several dates:
Manipulative and exploitative AI banned; organisations must support staff AI literacy.
Rules for model makers: documentation, copyright policy, training summary.
Disclose AI interaction and label deepfakes (Article 50).
Grace period ends for machine-readable marking; non-consensual intimate imagery banned.
Including recruitment and managing or evaluating workers (Annex III).
High-risk AI in products covered by EU safety law (Annex I).
What concerns designers most:
- AI literacy (Article 4), since February 2025: organisations that use AI must "take measures to support the development of AI literacy" of their staff (Art. 4). The 2026 amendment made it softer (no specific level is required), but it is still an obligation. A team learning path, like this course, is a concrete measure.
- Transparency (Article 50), since 2 August 2026: tell people when they're interacting with AI, and label deepfakes. Providers' machine-readable marking of AI content has a grace period to 2 December 2026 for systems already on the market (Commission guidelines; FAQ).
- High-risk uses (Annex III), from 2 December 2027: include AI for recruitment (for example CV or portfolio screening) and for allocating tasks or monitoring and evaluating workers (Annex III). If you design such a feature, it will need strict risk management, human oversight and documentation.
- General-purpose AI model rules, since August 2025, are mostly the concern of model makers (OpenAI, Anthropic, Google, Mistral…), not of designers using their tools.
4. Workers' rights
Introducing AI that changes how people work, or that could monitor them, often requires informing or consulting employees:
- EU-wide: before putting a high-risk AI system into use at the workplace, employers must inform workers' representatives and the affected workers (AI Act, Article 26(7)).
- Germany is a strict example: works councils co-determine the introduction of technical equipment intended to monitor behaviour or performance, must be informed in good time about planned work processes "including the use of AI", and may bring in an AI expert (BetrVG §87, §90, §80).
- Portugal requires employers to inform workers about the parameters, criteria and rules of algorithms that affect hiring, working conditions or monitoring (Lei 13/2023).
Ask your HR and legal teams what applies where your team and your users work.
5. Governance by design
Rules become real in the interface. When you design an AI feature:
- Transparency: say when people are talking to AI and when content is AI-generated. See AI for content design.
- Human oversight: approve, override and undo. Required for high-risk systems, good practice everywhere. See Human-in-the-loop and Agentic design.
- Logging: keep an audit trail that serves accountability without becoming surveillance of employees.
- Explanations fitted to each role: what an operator, a manager and an auditor need to know differs (NN/g).
6. Vocabulary
| Term | Meaning |
|---|---|
| Personal data | Any information about an identifiable person |
| Data protection officer (DPO) | The person responsible for data protection in an organisation |
| Data processing agreement (DPA) | The contract under which a vendor processes your data |
| DPIA | A data protection impact assessment, done before risky processing |
| Deployer | Under the AI Act, an organisation that uses an AI system |
| High-risk AI | Uses listed by the AI Act that need strict safeguards |
7. Try it (no code needed)
- List the AI tools you used last week, and for each one the kind of data you gave it.
- Classify that data with the four classes in section 2.
- For each tool, find out: is it on your organisation's approved list? Which plan? Is training on your content off?
- Write down one change you'll make (a different tool, anonymising first, a question for IT).
Reflection questions: Which of your research materials would you never paste into an AI tool? Does any feature you're designing allocate work to people or evaluate them?
8. If you lead a team
- Get the approved tools list and the data classes from IT security and the data protection officer; if there isn't one, help create it.
- Write one page of team AI usage guidelines and have it reviewed. See AI usage guidelines.
- Agree when to disclose AI use in deliverables (NN/g's PACED framework, adapted to design work).
- Involve employee representatives with HR before team-wide AI workflows.
- Keep a record of AI literacy activities as evidence for Article 4.
- Flag any feature that touches hiring, task allocation or performance as potentially high-risk.
Checklist
Learn more
- EU law: AI Act (consolidated) · Digital Omnibus on AI · Commission AI Act page · Article 50 guidelines · GDPR
- Unofficial but handy: AI Act Explorer and timeline (Future of Life Institute)
- NN/g: Crafting AI Explanations for Every Role · PACED
Books, courses and videos to go deeper
Related
AI usage guidelines · Copyright and AI-generated assets · Ethics, bias and inclusion in AI design · AI in UX research · AI tools landscape for designers · Agentic design · Personal data in prompts