Choose a Meta-Learner When Only 2 Percent of Users Were Treated
A growth team ran a push-notification test on 400,000 users. For operational reasons only 2 percent (8,000 users) received the push and 98 percent served as control; assignment was random with a known propensity of 0.02. Baseline 7-day activity is about 10 percent. They want to rank users by incremental activity and push the top 30 percent.
- Explain what goes wrong with a naive S-learner and with a plain T-learner here.
- Describe the X-learner steps and why the propensity enters the final blend. Which part of the model uses the large control group?
- The team asks "which learner should we ship?". Recommend an approach, including the final-stage model, a second alternative to compare against and the validation metric.
- How noisy will an uplift@30% measurement on a held-out slice be? Compute its standard error assuming the test slice keeps the 2/98 split and 10 percent activity in both arms.
1. Failure modes.
- S-learner: one model on (X,T) where only 2 percent of rows have T=1. The tree or penalized model has little incentive to split on T because the effect is small relative to outcome noise and almost every row is control, so the estimated effect is shrunk toward zero and often nearly constant; the ranking then mostly reflects baseline risk. This is the regularization bias of the S-learner.
- T-learner: \hat\mu_1 is fitted on only 8,000 rows while \hat\mu_0 sees 392,000, so \hat\tau=\hat\mu_1-\hat\mu_0 inherits a very noisy \hat\mu_1 (and a flexible model fitted to 8,000 binary outcomes at a 10 percent rate overfits). The difference mixes a precise term with a noisy one.
2. X-learner. (i) Fit \hat\mu_0 on controls and \hat\mu_1 on treated. (ii) Impute individual effects: for treated users \tilde D^1_i=Y_i-\hat\mu_0(X_i), for controls \tilde D^0_i=\hat\mu_1(X_i)-Y_i. (iii) Regress \tilde D^1 on X among treated to get \hat\tau_1(x), and \tilde D^0 among controls to get \hat\tau_0(x). (iv) Blend \hat\tau(x)=g(x)\hat\tau_0(x)+(1-g(x))\hat\tau_1(x) with g=e=0.02, so the blend is 98 percent the treated-side model \hat\tau_1. The large control group enters through \hat\mu_0, which is accurate with 392,000 rows and is subtracted from every treated outcome in \tilde D^1; the noisy \hat\mu_1 only matters for the 2 percent-weighted \hat\tau_0.
3. Recommendation. Use an X-learner (or the R- or DR-learner with e=0.02 known) whose final stage is a simple regularized model (a ridge or shallow GBM on a few engineered features), because only 8,000 treated users carry the effect signal. Compare it against a T-learner with a simple (logistic) base model. Choose with uplift@30% or Qini on a held-out randomized fold, with bootstrap intervals, and sanity-check the winner with a placebo permutation. In our simulation with about 1,000 treated users of 50,000 (5 seeds, scored against the known true effect), the X-learner with a ridge final stage had RMSE 0.0400 and correlation 0.707 with the true effect; X with a GBM final stage had 0.0751 and 0.454, and a GBM T-learner 0.0774 and 0.546 (verified). The learner label mattered less than the capacity of the stage that fits \tau. Flag that the simulated effect was nearly linear in the features, which favours ridge, so repeat on your data.
4. Noise of uplift@30%. The top 30 percent contains about 0.3\times8{,}000=2{,}400 treated and 0.3\times392{,}000=117{,}600 control users. With activity 10 percent in both arms the variance per user is 0.10\times0.90=0.09, so \text{SE}=\sqrt{0.09/2{,}400+0.09/117{,}600}\approx0.0062, i.e. 0.62 percentage points (verified). A real uplift of 2 points in the top 30 percent would be about 3 standard errors from zero, but distinguishing two models whose uplifts differ by half a point is hopeless on this holdout. The binding constraint is the treated count, so the best lever is a larger treatment share in the next randomized test (for example a 10 percent exploration slice), not a fancier learner.
Share this question