Marcus Andrey Study
Ler em português Library

UX Research · Step 5 · 9 min read

Analytics and A/B tests for designers

Analytics tells you what people do in your product, at scale; it can't tell you why. What analytics is good for, its traps, how an A/B test works, how to plan one before launch, why you mustn't peek, and where designers add the most value.

Analytics is a very good witness with no opinion about motives. It saw everyone who left the form at step three. It has no idea whether they were confused, interrupted, or just checking the price. An A/B test goes one step further: it can prove that a change caused a difference. It still can't tell you why. Designers who can read both, and pair them with the research that explains them, end up in the room where product decisions get made.

What you'll be able to do after reading:

  • Say which questions analytics can answer, and which it can't
  • Avoid the most common traps in reading product data
  • Plan an A/B test before launch: hypothesis, metrics, sample size, duration
  • Explain why peeking at a running test breaks it

1. What analytics is good for

Product analytics records what people do: pages and screens viewed, buttons pressed, where they came from and where they left. It shows behaviour at scale, from everyone, not from the five people in your study.

QuestionAnalytics can tell youIt can't tell youPair it with
Where do people drop out?62% leave the "add expense" form at the split stepWhether they were confused, interrupted or just checkingA usability test of that step
Is a feature used?Who opens it, how often, from whereWho needed it and never found itInterviews, or a short intercept survey
Which path is typical?The most common routes between screensWhat people were trying to do; two people on one path can want different thingsContextual inquiry or diary study
Did the change help?Before and after, but other things changed tooCause and effectAn A/B test, when there's enough traffic
Analytics shows what happened, at scale; research shows why. The example numbers are made up for the bill-splitting app.Diagram: Marcus Andrey, CC BY 4.0. After NN/g, Understanding User Pathways in Analytics (Laubheimer, 2022) and A/B Testing 101 (2024).

It's strong for three jobs:

  • Finding where to look. A funnel that loses 62% at one step tells you which step to put in front of a usability test.
  • Sizing a problem you found in research. Five people struggled with the split step in testing; analytics says how many people reach it each week.
  • Watching a change over time, with the caution that other things change at the same time.

And it's blind in one direction. NN/g's article on pathways puts it simply: analytics can't observe what's going on in users' heads, so two people with the exact same clicks may have had very different experiences (NN/g, Understanding User Pathways in Analytics, 2022). That's why the right column of the figure always names a research method. Using several methods to check one another is called triangulation (NN/g, Triangulation).


2. Traps in reading the numbers

  • Counting what's easy instead of what matters. Page views are easy to count. Did people get their job done? Google's HEART framework starts from goals, then signals, then metrics, for exactly this reason (Rodden, Hutchinson and Fu, CHI 2010); Defining success walks through it.
  • Averages that hide groups. A median session of 3 minutes can be half the people at 30 seconds and half at 6 minutes. Split by a meaningful segment before you conclude.
  • Before and after. "We shipped the new form and conversion went up 8%." So did the season, the marketing budget and the price. Before-and-after comparisons can't separate your change from everything else.
  • The tidy path. Path diagrams show the most common routes, aggregated from very different visits. NN/g suggests isolating one key page or screen and reading the traffic in and out of it, instead of the whole tangle.

3. What an A/B test is

An A/B test shows two or more versions of a design to live traffic at the same time, and splits visitors at random so each person sees one version. Then it compares a metric decided in advance (NN/g, A/B Testing 101, 2024). Version A is usually the current design, the control; B is the variant.

The random split is the whole point. Because the two groups are alike in everything except the version they saw, a difference between them, beyond what chance explains, was caused by the change. That's something analytics, surveys and interviews can't give you.

NN/g lists when not to use one:

  • Low traffic. You often need thousands of people to reach a reliable result.
  • Several changes at once. If B changes the copy, the layout and the colour, you won't know which mattered.
  • When you need the why. The test says B won; research says why.

4. Plan it before launch

A/B test plan: the payment reminder

Written before launch · bill-splitting app
Why

In interviews, 4 of 6 people said they wait to ask for money because asking feels rude, and that our reminder "sounds like me nagging".

Hypothesis

If the reminder speaks as the app, on a fixed monthly date, more people will pay within 7 days, because nobody has to be the one who asks.

The one change

A, control: "Ana says you owe €18.50. Pay now"

B, variant: "Monthly settle-up: €18.50 to Ana. Settle up"

Primary metric

Reminders followed by a payment within 7 days. Today: 3.0%

Guardrails
  • People who turn reminders off
  • Support contacts about reminders

Neither may get worse.

Smallest effect worth it

20% relative: from 3.0% to 3.6%

Sample and time

About 13,900 reminders per version at 95% significance and 80% power. With 2,000 reminders a day, split in half: 14 days, two full weeks.

Decision rule

Look at the result once, on day 14. Ship B if the primary metric is better and significant and no guardrail is worse. Otherwise keep A and write down what we learned.

A filled-in A/B test plan, written before launch so nobody can move the goalposts afterwards. The hypothesis comes from qualitative research; the numbers decide when the test ends.Diagram: Marcus Andrey, CC BY 4.0. Structure after NN/g, A/B Testing 101 (2024). Sample size computed for two proportions, 3.0% against 3.6%, two-sided α = 0.05, power 0.8.

NN/g's steps are the ones in the plan: a hypothesis grounded in research, one change, a primary metric and guardrail metrics that must not get worse, and a duration that comes from the sample size, not from patience (NN/g, 2024).

The sample size depends on three numbers:

  1. Baseline: today's value of the metric, from analytics. Here, 3.0%.
  2. Minimum detectable effect: the smallest change worth acting on. Here, 20% relative, so 3.6%. Smaller effects need far more traffic: NN/g shows that trying to detect a 1% relative change on a 3% baseline would need an enormous sample.
  3. Significance and power: usually 95% and 80%.

With those three, a standard sample-size calculator for two proportions gives about 13,900 per version. At 2,000 reminders a day, that's two weeks. NN/g recommends at least one to two weeks anyway, even with plenty of traffic, because behaviour changes across days of the week.

The guardrails protect you from a win that's really a loss. A variant that gets more clicks by being pushier might also get more people turning notifications off. NN/g warns against judging a test on a single metric for exactly this reason. Ron Kohavi, Diane Tang and Ya Xu, who ran experimentation at Microsoft, Google and LinkedIn, devote a chapter of their book to choosing an Overall Evaluation Criterion, the metric that really stands for long-term success (Cambridge University Press, 2020).


5. Don't peek, and don't believe the surprise

The most common way to break an A/B test is to watch the dashboard and stop when it turns green. The significance numbers assume the sample size was fixed in advance. In Evan Miller's example, if you check after every visitor and stop at the first "significant" result, a test of two versions that really perform the same declares a winner 26.1% of the time, instead of the 5% you think you're accepting (Evan Miller, How Not To Run an A/B Test). His advice: decide the sample size first, and wait. NN/g lists stopping too early among the most common mistakes.

Two more habits from people who run thousands of tests:

  • Run an A/A test first. Show the same design to both groups. If the tool reports a difference, the setup is wrong (NN/g, 2024).
  • Distrust results that look too good. Kohavi, Tang and Xu call it Twyman's law: any figure that looks interesting or different is usually wrong. A 40% lift from a label change is more likely a tracking bug than a breakthrough. Check before you celebrate.

And remember that statistically significant isn't the same as worth it. With enough traffic, a tiny difference becomes significant; whether it's worth building is a product question (Jeff Sauro, 10 Things to Know about A/B Testing, MeasuringU, 2012).


6. Where designers add the most value

You don't need to run the statistics. You do need to bring the parts the statistics can't:

  • The hypothesis. NN/g reports that only about one in seven A/B tests is a winner, and the odds are worse without a hypothesis from research. The plan above starts from interviews; that's the designer's contribution.
  • The variant. One clear change, designed well, accessible in both versions.
  • The why, afterwards. Whatever wins, watch five people use it. If B won for a reason you didn't expect, you've learned something bigger than the test.
  • The ethics. Testing a darker pattern against a fair one and shipping the winner is still shipping a darker pattern. Research ethics and consent applies to experiments too, and Kohavi, Tang and Xu give ethics its own chapter.

7. Common mistakes

  • Reading a before-and-after change as proof the design worked.
  • Running an A/B test on a page with a few hundred visits a week.
  • Changing three things in the variant.
  • One metric, no guardrails.
  • Stopping the test the day it turns green.
  • Shipping a 0.1% "significant" lift that costs a month of work.
  • Treating analytics as the answer instead of the place to start looking.

8. Vocabulary

Term Meaning
Funnel The share of people who get through each step of a flow
Segment A subgroup of users, such as new or returning
Control, variant The current version (A) and the new one (B)
Primary metric The one number the test is meant to move
Guardrail metric A number that must not get worse
Minimum detectable effect The smallest change the test is designed to detect
Statistical significance How unlikely a difference this big would be if the versions were the same
Power The chance of detecting a real effect of the size you planned for
A/A test Two identical versions, to check the setup
Triangulation Checking one finding with several methods

9. Try it (no tools needed)

  1. Pick one screen in a product you work on, or use often.
  2. Write the A/B plan in the figure for one change to it: hypothesis, the one change, primary metric, two guardrails.
  3. Ask: where would the hypothesis come from? If the honest answer is "a hunch", what research would turn it into evidence?

Reflection questions: What's the last number your team celebrated? Was it a before-and-after, or a controlled test?


10. If you already practise

  • Write the test plan, including the decision rule, before launch, and share it. It stops arguments about what the result means.
  • Bring qualitative evidence to every hypothesis, and a quick usability check to every winner.
  • Add a guardrail for accessibility and support contacts on any test that changes a flow.
  • Keep a log of all tests, winners and losers. Kohavi, Tang and Xu call it institutional memory; it's where a team's real learning lives.

Checklist

Learn more

Books, courses and videos to go deeper

Book links go to Amazon. As an Amazon Associate I earn from qualifying purchases, at no extra cost to you.

Defining success · Sample size and margin of error · Measuring usability · Writing insights that change decisions · Measuring AI features

Saved only on this device.

↑ ↓ to move · Enter to open · Esc to close