Advanced Experimentation: Interference, Marketplaces & Platforms
The boundary: ab-testing covers hypothesis setup, power and MDE, sample-ratio mismatch, peeking and sequential tests, CUPED and the delta method, novelty plots and the basics of holdouts. This subject starts where that one ends, and assumes you know all of it: it is about what to do when the independence assumption itself fails (interference), when randomizing users is the wrong design, when the effect you need is months away, and how the machinery of a large experimentation platform is built. Observational designs (difference-in-differences, synthetic control, IV) are taught in causal-inference-basics; here we only use them as the analysis half of geo experiments.
Interviews ask this as "how would you test a change to our matching algorithm / pricing / feed ranking?", "what is a switchback test and when would you use it?" and "the test says +10% but the holdout says +4%, why?". The senior answer names the shared resource that couples the units, picks a design that breaks the coupling, and prices the loss of power in numbers. Everything executable below was run (Python 3.12 with NumPy, SciPy and statsmodels, fixed seeds) and is marked "(verified)". Statements recalled from the literature without re-running are marked "(from memory)".
SUTVA and How It Breaks
The stable unit treatment value assumption says unit i's outcome depends only on its own assignment: Y_i(\mathbf{T}) = Y_i(T_i). Under it, a 50/50 user-level test estimates the effect you care about, the global effect of shipping to everyone: \tau = \frac{1}{n}\sum_i \big(Y_i(\mathbf{1}) - Y_i(\mathbf{0})\big). When units interfere, the experiment's treated-versus-control difference estimates something else, and no amount of sample size fixes it. The bias is structural, not noise.
| Setting | Shared thing that couples units | Typical bias of a user-level A/B |
|---|---|---|
| Social network | Messages, content, social proof | Control users receive treated users' output; effect usually biased toward zero for network-positive features, direction not guaranteed |
| Marketplace (rides, delivery, lodging, jobs) | Finite supply, inventory, drivers, listings | Treated demand takes supply from control; lift overstated |
| Ad auctions and budgets | One advertiser budget, one auction, competition | Treated ads spend budget faster and win auctions control would have won; arms trade off |
| Shared infrastructure | Caches, rate limits, CPU, delivery fleets | A hungry treatment slows control; harm hidden or lift inflated |
| Learning systems | A model retrained on pooled logs | Treated behaviour trains the model control also uses |
The cannibalization pattern. In a supply-constrained marketplace the treatment does not create demand out of nothing, it reallocates a fixed pot. The treated arm looks better than the truth, the control arm looks worse than a world with no treatment, and the difference between them exaggerates what shipping would do: the lift you measure is partly stolen from control. The worked example below measures a 10.0 percent measured lift against a 5.9 percent truth (verified).
graph LR
D1["Treated riders: higher request rate"] --> P["Shared driver pool: fixed supply"]
D0["Control riders: unchanged request rate"] --> P
P --> F["Lower fulfilment share for everyone"]
F --> M["Treated arm above truth, control arm below truth"]
The sign is not universal. If treatment creates a positive externality (a better feed makes treated users post more, and control friends enjoy it), control improves and the measured effect shrinks. The first question is always: what do the units share, and which way does the sharing push control?