Practice — Advanced Experimentation: Interference, Marketplaces & Platforms (6 questions)

You work on a food-delivery marketplace. A new dispatch/matching algorithm assigns orders to couriers using predicted prep times and batching. Offline replay says it should cut median delivery time. A colleague proposes: "randomize orders 50/50 between the old and new matcher and compare delivery time after a week."

1. Explain why order-level randomization is biased here, and in which direction.
2. Propose a valid design. State the randomization unit, the block length or cluster definition, how carryover is handled, and how you would randomize and balance time of day.
3. Choose primary, guardrail and diagnostic metrics (include at least one metric for each side of the market).
4. Describe the analysis (estimator, standard errors, variance reduction) and the sample-size reasoning.
5. Name two things you could not learn from this test and how you would cover them.

A retailer wants to test a new in-store checkout flow. Because staff training cannot be split inside a store, the randomization unit is the store. There are 60 stores, 30 per arm, with about 400 customers per store during the test window. The metric is purchase conversion, baseline 20 percent, and the smallest effect worth detecting is 1 percentage point. A pilot shows an intra-store correlation (ICC) of 0.02.

1. How many customers per arm would an iid (customer-level) test need at two-sided alpha 0.05 and 80 percent power?
2. Compute the design effect and the number of customers and stores per arm the real design needs. Is the 60-store plan adequate?
3. What minimum effect can the 60-store plan detect?
4. List three ways to improve the design that actually help, and one that does not.