A/B Testing & Online Experimentation

An A/B test is the one tool a product data scientist has that turns "users who saw the new checkout converted more" into "the new checkout caused more conversions". Everything else in the analytics stack — dashboards, funnels, cohort charts — describes what happened; only a randomised experiment isolates what a change did. That is why experimentation platforms sit at the centre of how large product companies ship, and why "walk me through how you'd test this feature" is one of the most common interview prompts for product/analytics data science roles.

The statistics of an A/B test are deliberately simple — a two-sample comparison of means or proportions. The difficulty is everything around the test: choosing what to measure, deciding what to randomise, sizing the test so a null result means something, keeping the test healthy while it runs, and resisting the dozen ways an analyst can talk themselves into a false positive after the fact. This subject covers that surrounding discipline. The mechanics of the tests themselves — z-tests, t-tests, p-values and confidence intervals — are covered in the Hypothesis Testing subject; here we use them, not derive them.

By the end you should be able to take a feature proposal, write the experiment plan (hypothesis, metrics, unit, sample size, duration, guardrails), run health checks on a live test, and read the results — including the awkward cases like "the metric moved but it isn't significant" and "we found a segment where it works".

Why Experiment At All: Correlation vs Causation

Observational product data is riddled with selection: users who adopt a new feature differ from users who don't. Power users try new things, and power users retain better, so "feature adopters retain 20% better" tells you almost nothing about the feature.

Randomisation breaks that link. If assignment to treatment is a coin flip independent of everything about the user, the two groups are, in expectation, identical on every covariate — the ones you measure and the ones you don't. Any systematic difference in outcome is then attributable to the treatment. That is the whole argument, and it is why an experiment is the gold standard for causal claims (the formal potential-outcomes framing lives in the Causal Inference Basics subject).

Observational:                        Randomised:
  user traits ──► adopts feature        coin flip ──► treatment
       │               │                                 │
       └──► outcome ◄──┘                        outcome ◄┘
  (confounded: traits drive both)     (traits balanced across arms)

Two practical consequences:

• If you can randomise, do. Every observational method (matching, diff-in-diff, IV) needs untestable assumptions an experiment does not.
• If you cannot randomise (legal, ethical, technical, or the change is company-wide), reach for the causal-inference toolkit — but be honest that the evidence is weaker.