Terms I Wish I’d Known Earlier: 8 A/B Testing Concepts for People Who Already Run Tests

I never thought my stats major would become useful. Here we are. The more I dig into experimentation, the more I find these ideas genuinely interesting, and the more I see smart teams trip over the same few of them. These are the eight I've learned the most from, and the ones I wish I'd known earlier.

A p-value of 0.05 doesn't mean there's a 5% chance your result is wrong. The number that matters is False Positive Risk: given a significant result, how likely is it that nothing actually happened? It depends on how often your ideas work. With 95% confidence, 80% power, and a 10% success rate, 22% of significant results are false positives ^[1].

Now the real world. In practice, almost no program holds to 95% consistently. Many drop to 80% to learn faster, and the confidence level you actually run at changes the odds a lot:

At 80% confidence, roughly half your “wins” are noise. Loosening the threshold can be a fine trade-off to learn faster, but make it knowingly.

Power is your chance of catching a real effect. MDE is the smallest effect your test can detect. Most teams set MDE backwards: they pick whatever makes the test length tolerable. Underpowered tests don't just miss real effects; the “wins” they do find are badly inflated. At 10% power, a significant result can overstate the true effect about 3.5x ^[2].

Traffic is the ceiling nobody can hire their way past. One team with high order values and low conversion told me they can only run one to three tests a month. Even at Amazon scale, tests need one to two weeks, and much longer for narrow customer groups. If a change is too small to detect with your traffic, testing it just burns a slot. Decide your MDE from business value first, then check whether your traffic can support it.

Checking your dashboard daily and stopping the moment you hit significance feels diligent. It's self-sabotage. Every look is another chance for noise to cross the line, so your real false-positive rate climbs far past the 5% you signed up for. Check 20 times and it can reach 40% or more ^[3].

The rule is simple: set the sample size, duration, and stopping rule before launch, and don't change course because of an early lead. If you need to watch continuously, use sequential testing, which is built for it. Otherwise, wait and look once.

This is the one I hear about most. The test comes back flat overall, so the team slices it: mobile, new users, loyalty members, signed-in. Mobile shows a lift. Ship it for mobile?

Not yet. Check 20 segments at 95% confidence and you have a 64% chance of at least one false win ^[B]. The test was also sized for all traffic, not for that segment. The fix: treat a segment result as a hypothesis, then resize the test and rerun it for that segment.

The demand is real, though. Plenty of large retailers can only see topline conversion and have no idea which customers drove a result. Segment insight is valuable. It just isn't evidence until it's rerun.

You planned a 50/50 split and got 52/48. It feels fine, but it isn't. SRM means your traffic split doesn't match the plan, which almost always points to a broken pipeline: bucketing, redirects, bot filtering, or logging. About 6% of Microsoft's experiments show SRM, and LinkedIn has reported around 10% for some experiment types ^[4].

Pair this with Twyman's Law: any result that looks surprising is usually wrong. When a test posts a giant lift, check SRM before you celebrate. If the check fails, you don't interpret the result. You find the bug.

Your OEC is the single metric that defines success, and it should predict long-term value, not short-term clicks. The trap I hear about most is conversion rate. Conversion can rise while average order value falls, and you've lost money on a “winning” test. That's why strong programs treat revenue per visitor as the primary metric, and fall back to conversion or funnel progress only when traffic can't support revenue per visitor's higher variance.

Pick your OEC and a few guardrail metrics before launch. Otherwise you'll optimize a local number and quietly hurt the business.

If traffic is your constraint, and for most teams it is, this is the most useful technique people underuse. CUPED uses each user's behavior before the experiment to strip out predictable noise. Eppo reports it lets teams finish experiments up to 65% faster ^[5].

The limit: it only works for users with history. It won't help much on a flow built for first-time visitors. For returning-user surfaces, it's close to free statistical power.

The variant that wins overstates its true impact. To clear the significance bar it needed a real effect and some lucky noise, and the luck doesn't ship. The curse is worst in underpowered tests, and some teams now shrink reported lifts to correct for it.

Then novelty wears off. Users react to change itself, and that bump fades. Microsoft researchers watched a redesigned button draw 28% more clicks at launch, then saw the lift shrink day after day as users got used to it ^[6]. Behavior also keeps shifting after a winner ships, and few teams can afford to keep monitoring it.

The result: your quarterly wins won't add up to what shows in the business. The most mature programs don't manage to win rate at all. They measure revenue actually attributed to tests, plus the “saves”: losing ideas that never reached customers.

The Pattern

Every concept here says the same thing: the dashboard number is evidence, not truth, and weaker evidence than we'd like. The best experimenters I've met aren't the ones with the fanciest stats. They've been burned by each of these once and built the habit to avoid it.

The dashboard number is evidence, not truth, and weaker evidence than we'd like.

Lately I've been interested in what happens before a test goes live: using synthetic users to catch obvious losers early, so scarce traffic goes to ideas that deserve it. It doesn't replace a live test. But when win rates sit around 10–20% ^[7] and traffic is the ceiling, deciding what earns a test slot matters more than ever.

Sources

1. Kohavi, R., Deng, A., & Vermeer, L. (2022). A/B Testing Intuition Busters: Common Misunderstandings in Online Controlled Experiments. KDD '22.
   False Positive Risk figures, published success rates.
2. Statistically significant results from low-power analyses: A comedy of errors (2026). ScienceDirect.
3. Statsig. Sequential testing: How to peek at A/B test results without ruining validity.
4. GrowthBook. Sample Ratio Mismatch (SRM): Types, Causes, and How to Identify.
5. Eppo. CUPED and CUPED++: Bending time in experimentation.
6. Sadeghi, S., et al. (2022). Novelty and Primacy: A Long-Term Estimator for Online Experiments. Technometrics.
7. Kohavi, R. Ronny Kohavi on teaching A/B testing at scale. Substack interview.