Marcus Andrey Study
Ler em português Library

UX Research · Step 1 · 6 min read

When not to do research

A study costs time, money, and other people's attention. How to tell when a new study is worth it, the signs that it isn't, what to do instead, and how to say "not now" without sounding like you don't care about users.

Knowing when not to run a study is part of the craft, not a betrayal of it. A study that can't change anything wastes participants' time, the team's patience and your credibility. The designers people trust with research aren't the ones who research everything. They're the ones who can say: "We already know enough here, ship it and watch this number," and "Here we really don't know, and it's expensive to be wrong."

What you'll be able to do after reading:

  • Judge whether a new study is worth it with two questions
  • Recognise the signs of research that can't change anything
  • Choose a lighter way to learn when a full study isn't needed
  • Say "not now" in a way that builds trust instead of losing it

1. Research has a cost

Even a small round of interviews costs a week or two of someone's time, the participants' time and incentives, and the team's attention while it waits. That's cheap compared with building the wrong thing, which is why research usually pays for itself. But "usually" is not "always". When the decision is small, easy to undo, or already made, the same week is better spent elsewhere.

There's a second cost that's easy to forget: every participant gives you their time. Asking people to help with a study nobody will act on isn't harmless. It's one of the reasons the ethics lesson that follows starts from respect for their time.


2. Two questions decide

Research first

A one-way door and little evidence. This is where a study pays for itself.

New product, pricing, a change people can't opt out of
Check what you know

Read the research and data you already have. Test only the riskiest assumption left.

A redesign of a flow you've studied before
Ship small and measure

Cheap to reverse, so the release is the study. Decide up front what you'll watch.

A label, a default, an empty state
Just decide

You know enough and it's easy to change. A study here is a delay with a nicer name.

Following your own design system
Two questions decide whether a study is worth it: how costly is a wrong decision to undo, and how much do you already know? Only one corner needs a new study before you decide. Top left: costly and little known, research first. Top right: costly but well known, check the evidence. Bottom left: easy to undo and little known, ship small and measure. Bottom right: easy to undo and well known, just decide.Diagram: Marcus Andrey, CC BY 4.0. "One-way" and "two-way doors" are from Jeff Bezos, 2015 letter to shareholders.
  1. How costly is it to be wrong? Jeff Bezos split decisions into "one-way doors", which are "irreversible or nearly irreversible" and need care, and "two-way doors", which are "changeable, reversible", where "you can reopen the door and go back through" (Amazon, 2015 letter to shareholders). A new pricing model is a one-way door. A button label is a two-way door.
  2. How much do you already know? Past studies, analytics, support tickets, sales calls and well-tested patterns are all evidence. Sometimes you know more than you think.

Only one corner of the matrix calls for a new study before deciding: costly to undo and little known. In the other three, there's a faster honest move.


3. Signs a study won't change anything

  • The decision is already made. If leadership has committed and the study can only confirm it, it's theatre. Ask: "What result would make us change course?" If the answer is "none", say so kindly and spend the time elsewhere.
  • Nobody owns the decision. No owner, no date, no action. Go back to Research questions that lead to decisions.
  • The answer already exists. NN/g notes that checking other sources "can be as simple as going to check your existing body of analytics or customer-satisfaction surveys" (Whitenton, Triangulation, NN/g, 2021). Look before you recruit.
  • The question is for a different method. "How many people use export?" is a query, not a study.
  • You can't reach the right people in time. Interviewing colleagues about a public product, or friends about a professional tool, can mislead more than it helps. Wait, change the question, or use a different source.
  • The risk to participants is bigger than the benefit. Sensitive topics and vulnerable groups need more care, not a rushed study. The next lesson covers this.

4. Lighter ways to learn

"Not a new study" doesn't mean "no evidence":

Situation Instead of a new study
A question past research may have answered Read the repository, or ask the person who ran the last study
"How many?" or "how often?" Analytics, logs, a support-ticket tag count
A small, reversible change Ship it to some users and watch one agreed number (see Analytics and A/B tests for designers)
A design that follows known patterns An expert review against heuristics (Heuristic evaluation)
You need a quick sense check Three short sessions with the right people (Test with three people)

Shipping and measuring is research too, if you decide beforehand what you'll watch and what would make you roll back. Without that, it's just shipping.


5. "Not now" is not "never"

NN/g's answer to "when should I do research?" is "at whatever stage you're in right now", and as early as possible, because early findings have the most impact (Farrell, UX Research Cheat Sheet, NN/g, 2017). That's not a contradiction. The point of this lesson isn't to research less; it's to put the research where it changes things.

When you say no, offer the alternative in the same breath:

"We don't need a new study for this: the last round already showed people miss the filter, and the change is easy to undo. Let's ship it and watch filter use for two weeks. I'd rather spend the interviews on the pricing question, where we really don't know."

That sentence shows you care about users, about the team's time and about the decision. It's how research earns a seat at the table.


6. Common mistakes

  • Researching everything, so the team learns to route around research.
  • Running a study after the decision, to make it look evidence-based.
  • Recruiting whoever is available because the study "has to happen this week".
  • Shipping without agreeing what to measure, then calling it an experiment.
  • Saying no without offering a lighter way to learn.

7. Vocabulary

Term Meaning
One-way door A decision that's very costly or impossible to undo
Two-way door A decision you can reverse cheaply if it's wrong
Research theatre A study run for show, when no result could change the decision
Desk research Learning from what already exists: past studies, data, reports
Expert review Checking a design against known principles, without participants

8. Try it (no tools needed)

  1. List four decisions your team will make in the next month.
  2. Place each one on the matrix: how costly to undo, how much you already know.
  3. For the one in the top-left corner, write the research question. For the others, write the lighter way to learn.

Reflection questions: Is there a study on your roadmap that no result could change? What would happen if you said so?


9. If you already practise

  • Add "What result would change our course?" to your intake for research requests. It filters theatre early.
  • Keep a short list of "decisions we chose not to research, and why". It shows judgement, and it's useful when one of them goes wrong.
  • When you ship instead of study, write the rollback rule next to the metric.

Checklist

Learn more

Books, courses and videos to go deeper

Book links go to Amazon. As an Amazon Associate I earn from qualifying purchases, at no extra cost to you.

Research questions that lead to decisions · Choosing a research method · Heuristic evaluation · Test with three people · Analytics and A/B tests for designers · Research repositories

Saved only on this device.

↑ ↓ to move · Enter to open · Esc to close