Design the Switchback for a Surge Multiplier Test
A ride-hailing company wants to know whether a new surge policy raises profit per session by at least 2.5 percent. The city has 8 pricing zones, about 95,000 sessions per day, and drivers who take roughly an hour to respond to a price change. The product manager proposes randomizing 50 percent of riders to the new policy for a week.
- Explain why the rider-level test is the wrong design here, in terms of what it would measure.
- Specify a switchback: the randomization unit, block length, burn-in, assignment scheme, analysis estimator and standard error, and what you would run beforehand to check the design.
- Estimate the duration, and explain why a simple per-arm sample-size calculation would be too optimistic.
1. Why the rider-level test fails. Treated and control riders share one pool of drivers, and drivers respond to the blended price they see. Treated riders who drop out free cars for control riders, and supply moves only about halfway toward where it would go if everyone paid the higher price. In the synthetic simulation behind this lesson, a 50/50 rider-level test of a 1.15x multiplier recovered +2.1 percent profit per session, where the true global effect was +13.8 percent: it recovered 16 percent of the effect and overstated the trip loss by 2.5 times. The test does not estimate the policy's effect on a market where everyone gets it.
2. Design. Randomize (zone, time block) cells, so the whole local market switches together.
- Block length: long relative to supply's response time. With a one-hour driver response I would use 8-hour blocks (the simulation showed bias near zero at 8 hours with burn-in and a large bias toward zero at 30 to 60 minutes).
- Burn-in: drop the first 2 hours of each block (about two response times in my setup; the simulation needed burn-in covering several supply time constants). Pre-register it.
- Assignment: random per block with balance within each day, a random per-zone offset of the block grid so blocks do not always fall on the same clock hours, complete blocks only.
- Estimator: per-arm ratio of sums (total profit over total sessions), difference between arms, with block-clustered standard errors; adjust or stratify for zone and hour of day.
- Pre-checks: an A/A switchback on historical data or a simulator to verify the false-positive rate (in the simulation, a naive session-level analysis gave 52 percent false positives at a nominal 5 percent; the clustered analysis gave 8.7 percent, still a touch liberal, so consider a t reference), and a power calculation with the design effect.
3. Duration. A per-session calculation (about 37,000 sessions per arm for a 0.02 difference on a 0.787 baseline in the simulation) says under one day of traffic. That ignores the fact that sessions in the same block share supply and demand shocks: the design effect was about 10.8, so 943,000 sessions carried the information of about 87,000 independent ones. Properly sized, detecting a 2.5 percent effect took roughly 15 days in the simulation. Plan for two to three weeks and cover full weekly cycles; the numbers are from a synthetic market and your own historical data should replace them.
Share this question