Design an Experiment for a New Matching Algorithm in a Two-Sided Delivery Marketplace
You work on a food-delivery marketplace. A new dispatch/matching algorithm assigns orders to couriers using predicted prep times and batching. Offline replay says it should cut median delivery time. A colleague proposes: "randomize orders 50/50 between the old and new matcher and compare delivery time after a week."
- Explain why order-level randomization is biased here, and in which direction.
- Propose a valid design. State the randomization unit, the block length or cluster definition, how carryover is handled, and how you would randomize and balance time of day.
- Choose primary, guardrail and diagnostic metrics (include at least one metric for each side of the market).
- Describe the analysis (estimator, standard errors, variance reduction) and the sample-size reasoning.
- Name two things you could not learn from this test and how you would cover them.
1. Why order-level randomization is biased. Both matchers draw on the same pool of couriers. If the new matcher is better at grabbing the nearest courier or at batching, it takes couriers that control orders would otherwise have received, so control deliveries get slower while treated ones get faster. The measured gap is partly a transfer between arms, not an improvement for the city: shipping to 100 percent removes the pool the gain was taken from. For a supply-constrained marketplace the measured lift is therefore overstated (in our simulated ride-hailing market a rider-level test showed +10.0 percent against a true +5.9 percent; verified). It is also interference at the courier level: one courier serves both arms, so assignments in one arm change that courier's state for the other.
2. Design. The coupling is global within a city (one courier pool), so use a switchback: randomize whole-city time blocks. Or, if the company has many independent cities, run a geo-level (cluster) experiment with cities as clusters; with only a handful of comparable cities that has little power, so the switchback (many blocks within each city) is the default and a geo holdout is the later confirmation.
- Unit: a city-by-time block. Randomize the algorithm for the entire block.
- Block length: long enough that the effect of the previous block has washed out, short enough to have many blocks. The relevant timescale is the order lifecycle plus courier repositioning: prep 15-25 minutes, delivery 20-30 minutes, so state from the previous block lingers for roughly half an hour. A 2-hour block with the first 30 minutes discarded (burn-in) is a sensible starting point; in the simulated market this had bias 0.05 points against 1.36 points for 10-minute blocks (verified).
- Randomization: randomize adjacent block pairs (one old, one new, order flipped at random) so every day-part and weekday is balanced; alternatively stratify by day-part. Keep orders in flight at a block boundary attributed to the algorithm that assigned them.
- Carryover check: estimate the effect separately for the early and late parts of treated blocks; a difference reveals carryover, and a placebo switchback (both arms the old matcher) should show nothing.
3. Metrics. Primary: median and p90 delivery time per order or completed orders per active courier-hour (pick one OEC in advance). Customer-side guardrails: order cancellation rate, late-delivery share, refund/support contact rate. Courier-side guardrails: earnings per active hour, idle time, acceptance/decline rate, courier churn (slow). Diagnostics: batching rate, assignment distance, matching latency, courier utilisation. The courier metrics are not optional: a matcher that speeds up deliveries by overworking couriers is a loss.
4. Analysis and sizing. The analysis unit is the block, not the order. Compute the metric per block (ratio of sums), compare treated and control blocks with a model that includes hour-of-week and city fixed effects for precision, and use block-clustered or permutation inference (re-randomize the block labels). Orders inside a block share the courier pool, so they are correlated: with a within-block ICC of, say, 0.002 and 900 orders per block the design effect is 1 + 899 \times 0.002 \approx 2.8 and the experiment needs about that many times the iid sample; a naive order-level z-test false-positived 15.2 percent of the time in our simulation at a nominal 5 percent (verified). Plan the number of blocks (power grows with blocks, not with orders per block), and round to whole weeks so every weekday is represented equally.
5. What you cannot learn. (a) Slow effects: courier retention and customer repeat behaviour change over months, so confirm with a staged geo rollout and a long-running holdout city; (b) the effect at full scale on a two-sided equilibrium (couriers adapt, restaurants change prep behaviour), which only a market-wide rollout with before/after or synthetic-control analysis can show. Also be explicit that heterogeneity by city size or time of day is exploratory unless pre-registered.
Share this question