UX Research · Step 5 · 9 min read
Analytics and A/B tests for designers
Analytics tells you what people do in your product, at scale; it can't tell you why. What analytics is good for, its traps, how an A/B test works, how to plan one before launch, why you mustn't peek, and where designers add the most value.
Analytics is a very good witness with no opinion about motives. It saw everyone who left the form at step three. It has no idea whether they were confused, interrupted, or just checking the price. An A/B test goes one step further: it can prove that a change caused a difference. It still can't tell you why. Designers who can read both, and pair them with the research that explains them, end up in the room where product decisions get made.
What you'll be able to do after reading:
- Say which questions analytics can answer, and which it can't
- Avoid the most common traps in reading product data
- Plan an A/B test before launch: hypothesis, metrics, sample size, duration
- Explain why peeking at a running test breaks it
1. What analytics is good for
Product analytics records what people do: pages and screens viewed, buttons pressed, where they came from and where they left. It shows behaviour at scale, from everyone, not from the five people in your study.
| Question | Analytics can tell you | It can't tell you | Pair it with |
|---|---|---|---|
| Where do people drop out? | 62% leave the "add expense" form at the split step | Whether they were confused, interrupted or just checking | A usability test of that step |
| Is a feature used? | Who opens it, how often, from where | Who needed it and never found it | Interviews, or a short intercept survey |
| Which path is typical? | The most common routes between screens | What people were trying to do; two people on one path can want different things | Contextual inquiry or diary study |
| Did the change help? | Before and after, but other things changed too | Cause and effect | An A/B test, when there's enough traffic |
It's strong for three jobs:
- Finding where to look. A funnel that loses 62% at one step tells you which step to put in front of a usability test.
- Sizing a problem you found in research. Five people struggled with the split step in testing; analytics says how many people reach it each week.
- Watching a change over time, with the caution that other things change at the same time.
And it's blind in one direction. NN/g's article on pathways puts it simply: analytics can't observe what's going on in users' heads, so two people with the exact same clicks may have had very different experiences (NN/g, Understanding User Pathways in Analytics, 2022). That's why the right column of the figure always names a research method. Using several methods to check one another is called triangulation (NN/g, Triangulation).
2. Traps in reading the numbers
- Counting what's easy instead of what matters. Page views are easy to count. Did people get their job done? Google's HEART framework starts from goals, then signals, then metrics, for exactly this reason (Rodden, Hutchinson and Fu, CHI 2010); Defining success walks through it.
- Averages that hide groups. A median session of 3 minutes can be half the people at 30 seconds and half at 6 minutes. Split by a meaningful segment before you conclude.
- Before and after. "We shipped the new form and conversion went up 8%." So did the season, the marketing budget and the price. Before-and-after comparisons can't separate your change from everything else.
- The tidy path. Path diagrams show the most common routes, aggregated from very different visits. NN/g suggests isolating one key page or screen and reading the traffic in and out of it, instead of the whole tangle.
3. What an A/B test is
An A/B test shows two or more versions of a design to live traffic at the same time, and splits visitors at random so each person sees one version. Then it compares a metric decided in advance (NN/g, A/B Testing 101, 2024). Version A is usually the current design, the control; B is the variant.
The random split is the whole point. Because the two groups are alike in everything except the version they saw, a difference between them, beyond what chance explains, was caused by the change. That's something analytics, surveys and interviews can't give you.
NN/g lists when not to use one:
- Low traffic. You often need thousands of people to reach a reliable result.
- Several changes at once. If B changes the copy, the layout and the colour, you won't know which mattered.
- When you need the why. The test says B won; research says why.
4. Plan it before launch
A/B test plan: the payment reminder
In interviews, 4 of 6 people said they wait to ask for money because asking feels rude, and that our reminder "sounds like me nagging".
If the reminder speaks as the app, on a fixed monthly date, more people will pay within 7 days, because nobody has to be the one who asks.
A, control: "Ana says you owe €18.50. Pay now"
B, variant: "Monthly settle-up: €18.50 to Ana. Settle up"
Reminders followed by a payment within 7 days. Today: 3.0%
- People who turn reminders off
- Support contacts about reminders
Neither may get worse.
20% relative: from 3.0% to 3.6%
About 13,900 reminders per version at 95% significance and 80% power. With 2,000 reminders a day, split in half: 14 days, two full weeks.
Look at the result once, on day 14. Ship B if the primary metric is better and significant and no guardrail is worse. Otherwise keep A and write down what we learned.
NN/g's steps are the ones in the plan: a hypothesis grounded in research, one change, a primary metric and guardrail metrics that must not get worse, and a duration that comes from the sample size, not from patience (NN/g, 2024).
The sample size depends on three numbers:
- Baseline: today's value of the metric, from analytics. Here, 3.0%.
- Minimum detectable effect: the smallest change worth acting on. Here, 20% relative, so 3.6%. Smaller effects need far more traffic: NN/g shows that trying to detect a 1% relative change on a 3% baseline would need an enormous sample.
- Significance and power: usually 95% and 80%.
With those three, a standard sample-size calculator for two proportions gives about 13,900 per version. At 2,000 reminders a day, that's two weeks. NN/g recommends at least one to two weeks anyway, even with plenty of traffic, because behaviour changes across days of the week.
The guardrails protect you from a win that's really a loss. A variant that gets more clicks by being pushier might also get more people turning notifications off. NN/g warns against judging a test on a single metric for exactly this reason. Ron Kohavi, Diane Tang and Ya Xu, who ran experimentation at Microsoft, Google and LinkedIn, devote a chapter of their book to choosing an Overall Evaluation Criterion, the metric that really stands for long-term success (Cambridge University Press, 2020).
5. Don't peek, and don't believe the surprise
The most common way to break an A/B test is to watch the dashboard and stop when it turns green. The significance numbers assume the sample size was fixed in advance. In Evan Miller's example, if you check after every visitor and stop at the first "significant" result, a test of two versions that really perform the same declares a winner 26.1% of the time, instead of the 5% you think you're accepting (Evan Miller, How Not To Run an A/B Test). His advice: decide the sample size first, and wait. NN/g lists stopping too early among the most common mistakes.
Two more habits from people who run thousands of tests:
- Run an A/A test first. Show the same design to both groups. If the tool reports a difference, the setup is wrong (NN/g, 2024).
- Distrust results that look too good. Kohavi, Tang and Xu call it Twyman's law: any figure that looks interesting or different is usually wrong. A 40% lift from a label change is more likely a tracking bug than a breakthrough. Check before you celebrate.
And remember that statistically significant isn't the same as worth it. With enough traffic, a tiny difference becomes significant; whether it's worth building is a product question (Jeff Sauro, 10 Things to Know about A/B Testing, MeasuringU, 2012).
6. Where designers add the most value
You don't need to run the statistics. You do need to bring the parts the statistics can't:
- The hypothesis. NN/g reports that only about one in seven A/B tests is a winner, and the odds are worse without a hypothesis from research. The plan above starts from interviews; that's the designer's contribution.
- The variant. One clear change, designed well, accessible in both versions.
- The why, afterwards. Whatever wins, watch five people use it. If B won for a reason you didn't expect, you've learned something bigger than the test.
- The ethics. Testing a darker pattern against a fair one and shipping the winner is still shipping a darker pattern. Research ethics and consent applies to experiments too, and Kohavi, Tang and Xu give ethics its own chapter.
7. Common mistakes
- Reading a before-and-after change as proof the design worked.
- Running an A/B test on a page with a few hundred visits a week.
- Changing three things in the variant.
- One metric, no guardrails.
- Stopping the test the day it turns green.
- Shipping a 0.1% "significant" lift that costs a month of work.
- Treating analytics as the answer instead of the place to start looking.
8. Vocabulary
| Term | Meaning |
|---|---|
| Funnel | The share of people who get through each step of a flow |
| Segment | A subgroup of users, such as new or returning |
| Control, variant | The current version (A) and the new one (B) |
| Primary metric | The one number the test is meant to move |
| Guardrail metric | A number that must not get worse |
| Minimum detectable effect | The smallest change the test is designed to detect |
| Statistical significance | How unlikely a difference this big would be if the versions were the same |
| Power | The chance of detecting a real effect of the size you planned for |
| A/A test | Two identical versions, to check the setup |
| Triangulation | Checking one finding with several methods |
9. Try it (no tools needed)
- Pick one screen in a product you work on, or use often.
- Write the A/B plan in the figure for one change to it: hypothesis, the one change, primary metric, two guardrails.
- Ask: where would the hypothesis come from? If the honest answer is "a hunch", what research would turn it into evidence?
Reflection questions: What's the last number your team celebrated? Was it a before-and-after, or a controlled test?
10. If you already practise
- Write the test plan, including the decision rule, before launch, and share it. It stops arguments about what the result means.
- Bring qualitative evidence to every hypothesis, and a quick usability check to every winner.
- Add a guardrail for accessibility and support contacts on any test that changes a flow.
- Keep a log of all tests, winners and losers. Kohavi, Tang and Xu call it institutional memory; it's where a team's real learning lives.
Checklist
Learn more
- Tim Neusesser, A/B Testing 101 (NN/g, 2024)
- Evan Miller, How Not To Run an A/B Test
- Page Laubheimer, Understanding User Pathways in Analytics (NN/g, 2022)
- Kerry Rodden, Hilary Hutchinson and Xin Fu, Measuring the user experience on a large scale (CHI 2010): the HEART framework
- Jeff Sauro, 10 Things to Know about A/B Testing (MeasuringU, 2012)
- Experiment Guide, the companion site of Kohavi, Tang and Xu's book
Books, courses and videos to go deeper
- Trustworthy Online Controlled Experiments: A Practical Guide to A/B TestingRon Kohavi, Diane Tang and Ya XuBook · Cambridge University Press, 2020 (Amazon)
- Quantifying the User Experience: Practical Statistics for User ResearchJeff Sauro and James R. LewisBook · Morgan Kaufmann, 2nd edition, 2016 (Amazon)
Book links go to Amazon. As an Amazon Associate I earn from qualifying purchases, at no extra cost to you.
Related
Defining success · Sample size and margin of error · Measuring usability · Writing insights that change decisions · Measuring AI features