Why Adopters Look Better Than They Are
A "smart playlists" feature shipped to everyone last quarter. Users who turned it on have 30-day retention of 72%; users who did not have 55%. A PM writes "smart playlists lift retention by 17 points" in the quarterly review.
- Write the naive difference as a decomposition into a causal quantity and a bias term, and explain in words what the bias term is.
- Name the causal estimand the PM actually cares about, and give a concrete reason the bias term is probably positive here.
- Sketch, in one paragraph, an observational analysis you would run instead, listing the assumption it needs and one check you would do.
1. Decomposition
$E[Y \mid D=1] - E[Y \mid D=0] = \underbrace{E[Y(1)-Y(0) \mid D=1]}_{\text{ATT}}
- \underbrace{E[Y(0) \mid D=1] - E[Y(0) \mid D=0]}_{\text{selection bias}}$.
The bias term is the difference in untreated retention between the people who chose to adopt and those who did not — how much better adopters would have retained even without the feature. It is a counterfactual for adopters, so it is not observable directly; that is why the naive comparison cannot be interpreted causally.
2. Estimand and sign
The PM cares about the ATT: what did the feature do for the users who adopted it (or, for a rollout decision, the ATE for the marginal user who would adopt with better placement). The bias is very likely positive: adoption requires opening settings and exploring, which is what engaged users do, and engaged users retain better regardless. Adopters also had to survive long enough to find the feature — a form of conditioning on the outcome. Both push the naive 17 points upward.
3. An observational analysis
Restrict to users active in the four weeks before launch, take pre-launch covariates only (sessions, tenure, plan, prior retention history, platform), and estimate the ATT with a doubly-robust estimator (propensity model plus outcome model). Assumption: no unmeasured factor drives both adoption and retention once these covariates are held fixed, and there is overlap (some non-adopters look like adopters). Checks: standardised mean differences after weighting below 0.1; a placebo outcome such as retention in the month before launch, which the feature cannot have caused — if adopters still "retain better" on the placebo, residual confounding remains; and an E-value stating how strong a hidden confounder would need to be to erase the effect. State up front that the honest fix is a holdout in the next launch.
Share this question