Compute the Design Effect and the Number of Stores Needed in a Store-Level Experiment
A retailer wants to test a new in-store checkout flow. Because staff training cannot be split inside a store, the randomization unit is the store. There are 60 stores, 30 per arm, with about 400 customers per store during the test window. The metric is purchase conversion, baseline 20 percent, and the smallest effect worth detecting is 1 percentage point. A pilot shows an intra-store correlation (ICC) of 0.02.
- How many customers per arm would an iid (customer-level) test need at two-sided alpha 0.05 and 80 percent power?
- Compute the design effect and the number of customers and stores per arm the real design needs. Is the 60-store plan adequate?
- What minimum effect can the 60-store plan detect?
- List three ways to improve the design that actually help, and one that does not.
1. iid sample size. With z_{1-\alpha/2} + z_{1-\beta} = 1.96 + 0.84 = 2.80 (2.8016 exactly):
2. Design effect. With m = 400 customers per store and \rho = 0.02:
The clustered design needs 25{,}116 \times 8.98 \approx 225{,}500 customers per arm, i.e. 225{,}545 / 400 \approx 564 stores per arm (1,128 stores in total). The 60-store plan is nowhere near adequate. The 30 stores per arm contain 12,000 customers, but their effective sample size is 12{,}000 / 8.98 \approx 1{,}336 customers.
3. Detectable effect with 30 stores per arm.
That is a 21.7 percent relative lift on a 20 percent baseline, far more than the 1 point that matters. (All of these figures are arithmetic checked with code; the design-effect formula itself matched a cluster simulation: 10.93 empirical against 10.95 predicted at ICC 0.05 and m = 200.)
4. What helps and what does not. Helps: (a) more clusters if there are any: when m\rho \gg 1 the stores needed are about n_{\text{iid}}\rho = 25{,}116 \times 0.02 \approx 502 per arm regardless of store size, so only more stores (or more independent units, such as departments) move power; (b) reduce the effective ICC with covariate adjustment at the store level (adjust for the store's pre-period conversion, or match stores into pairs and randomize within pairs), which removes between-store variance before it enters the comparison; (c) change the question to a larger MDE the business would actually act on, or switch to a design with more units (randomize registers or lanes, or use a stepped-wedge rollout in which each store is its own control over time and analyse with store fixed effects and a rollout-order permutation test). Does not help: running longer or recruiting more customers per store. Going from 400 to 4,000 customers per store raises m roughly tenfold and the design effect with it (1 + 3{,}999 \times 0.02 = 81); you add data but almost no information, because the stores are still only 30 per arm.
Share this question