Intermediate
Open
Pro
Build and Read a Qini Curve From Decile Counts
A randomized holdout of 10,000 customers (5,000 treated, 5,000 control) was scored by an uplift model and split into ten score deciles of 1,000 customers each (500 treated and 500 control in every decile). Conversions per decile, best-scored decile first:
| Decile | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
|---|---|---|---|---|---|---|---|---|---|---|
| Treated conversions (of 500) | 120 | 100 | 90 | 75 | 65 | 60 | 55 | 50 | 45 | 40 |
| Control conversions (of 500) | 60 | 60 | 60 | 60 | 60 | 60 | 60 | 60 | 60 | 60 |
- Compute uplift in the top decile and the cumulative incremental conversions Q(k)=n_k(\bar Y_{T,k}-\bar Y_{C,k}) at 10, 30, 50 and 100 percent targeted.
- Draw the random-targeting line and compute the gap to the curve at each of those points. Where is the curve highest and what does that say about treating the bottom half?
- A colleague reports "our model's AUC for predicting conversion among treated customers is 0.74, so it works". Why is this not evidence, and what would you report instead?
- How would you put an error bar on the Qini coefficient, and what does a placebo test add?
Share this question