Only One Percent of Last Year's Customers Were Randomized
The only randomized data you have is a past campaign where 1 percent of 200,000 customers (about 2,000) received a coupon at random and 99 percent did not. A colleague wants to build a T-learner with two gradient-boosted models, one per arm, "because it is the standard".
In a controlled simulation (same population, 140,000 users for training, 60,000 for testing, average of three seeds), the true EUR value per user of each learner's top-15-percent audience was:
| Treated share | S | T | X | DR | Response model |
|---|---|---|---|---|---|
| 1% | 2.54 | 2.52 | 3.23 | 3.01 | -3.88 |
| 5% | 3.40 | 3.50 | 3.69 | 3.73 | -3.86 |
| 50% | 3.86 | 3.92 | 3.94 | 3.95 | -3.57 |
The oracle (true effect known) earns EUR 4.21 per user.
- Which learner would you pick for the 1 percent log, and why does the T-learner struggle there?
- How would you evaluate and compare the candidates when there are only a few hundred treated users in any holdout?
- What would you do differently in the next cycle so you are not stuck at 1 percent forever?
- Why does the response model not improve as the treated share grows?
1. Pick and reason. Prefer the X-learner, or a DR learner with a heavily regularized final stage, over the T-learner. In the 1 percent column the X-learner reaches EUR 3.23 per user (77 percent of the oracle), the DR 3.01, the T-learner 2.52 and the S-learner 2.54. The T-learner fits one flexible model on roughly 1,400 treated rows, whose errors are large; subtracting a noisy treated model from a good control model gives noisy effects, and the ranking inherits that noise. The X-learner imputes individual effects with the control model (fit on the 99 percent), regresses them within each arm and blends with weight near the propensity, so it leans on the abundant group. The DR pseudo-outcome divides by the propensity, so with e=0.01 a treated buyer weighs 100 times a control buyer: it is unbiased but high-variance unless the final stage is strongly regularized. At 5 percent the learners sit within a few cents of each other and at 50 percent they converge, so the choice matters exactly when treated users are scarce.
2. Evaluation with few treated. Do not use a single holdout. Use 5-fold cross-fitting so every user is scored by a model that never saw them, giving all 2,000 treated users an out-of-fold score. Then compare Qini/AUUC with bootstrap intervals; expect the intervals to overlap for close competitors (X, DR and a causal forest overlapped almost entirely in the 5 percent case), and choose on simplicity and stability, not a 2-point gap. Add a placebo test (permute treatment labels and confirm the Qini centres on zero with a range of about plus or minus 500 in the 5 percent case) to show the model is not fitting noise. Use coarse deciles or fewer segments because a segment of 2,000 customers contains only 20 treated.
3. Next cycle. The randomization share is a design choice with a known price. A 50/50 user carries e(1-e)=0.25 of information against 0.0099 at 1 percent, about 25 users of the old log. A reserved exploration slice of 5 to 10 percent of the base (randomized 50/50) cost about EUR 7,400 to 14,900 per cycle in the simulation. Randomizing only users near the cutoff is cheaper but needs the propensity logged carefully. Log scores, propensity and arm for every user.
4. The response model. It optimizes the wrong target: predicting purchase ranks sure things highest, and sure things have no (or negative) effect. More randomized data helps estimate effects; it cannot turn a conversion predictor into an effect predictor. It stays at about -EUR 3.9 per user because it keeps choosing customers who would buy anyway and who then also redeem the discount.
Share this question