Advanced Experimentation: Interference, Marketplaces & Platforms
Experiments that survive network effects, two-sided markets and long horizons
Run experiments when units are not independent: how SUTVA breaks in social networks, marketplaces, ad auctions and shared infrastructure, why a rider-level test overstates a marketplace lift, how to diagnose interference with exposure mappings and neighbour regressions, and the designs that fix it (cluster randomization, ego-clusters, switchback with carryover and burn-in, budget and supply splits, geo experiments, staggered rollouts) with the design effect 1 + (m - 1) ICC, cluster-robust and permutation analysis, long-term effects (universal holdouts and their contamination, the surrogate index), heterogeneity without p-hacking, team-draft interleaving, non-inferiority tests, and the anatomy of an experimentation platform (hashing and salts, layers, triggered analysis). Includes a ride-hailing worked example with simulated numbers for bias, switchback block length and sample size.
Practice questions (6)
-
View →
Design an Experiment for a New Matching Algorithm in a Two-Sided Delivery Marketplace
Advanced · Free -
View →
Compute the Design Effect and the Number of Stores Needed in a Store-Level Experiment
Intermediate · Free -
View →
Detect and Handle Interference in a Social-Feed Experiment
Advanced -
View →
Plan a Long-Term Holdout Program for a Growth Team
Advanced -
View →
Critique an Interleaving Proposal for a Search Team
Advanced