Compute and Interpret PSI for a Drifting Feature
A credit-risk model uses monthly_income bucketed into 4 quantile bins
defined on the training set (each holding 25% of training rows). This
week's production distribution across the same bins is:
| Bin | Expected (train) | Actual (this week) |
|---|---|---|
| 1 (lowest) | 0.25 | 0.40 |
| 2 | 0.25 | 0.30 |
| 3 | 0.25 | 0.20 |
| 4 (highest) | 0.25 | 0.10 |
- Compute the PSI for this feature. Show the per-bin contributions.
- Using the common rule-of-thumb thresholds, what does the value mean and what would you do?
- Your colleague argues that a KS test on the raw values would be
"more rigorous" and proposes alerting on
p < 0.05. What is the problem with that at production scale (millions of rows per day)?
1. PSI calculation
PSI = Σ (A_i − E_i) · ln(A_i / E_i)
| Bin | A − E | ln(A/E) | Contribution |
|---|---|---|---|
| 1 | +0.15 | ln(1.60) = 0.470 | 0.0705 |
| 2 | +0.05 | ln(1.20) = 0.182 | 0.0091 |
| 3 | −0.05 | ln(0.80) = −0.223 | 0.0112 |
| 4 | −0.15 | ln(0.40) = −0.916 | 0.1374 |
PSI ≈ 0.0705 + 0.0091 + 0.0112 + 0.1374 ≈ 0.228.
Note that every contribution is non-negative (the difference and the log always share a sign) and that the bin losing mass (bin 4) contributes the most, because the log ratio ln(0.4) is large in magnitude.
2. Interpretation and action
Using the usual bands (< 0.10 no significant change, 0.10–0.25 moderate, > 0.25 significant), 0.228 is a moderate shift, close to the significant line. The population has moved sharply toward lower incomes. This is a warn/ticket-tier signal, not a page: it should be investigated within the day, correlated with deploy markers and upstream changes (did an income normalisation or currency step change? did a new acquisition channel launch?), and checked against prediction drift (has the approval rate moved?). If the shift is genuine — a new customer segment — it strengthens the case for a retrain on recent data. If it is a pipeline artefact, fix the pipeline; do not retrain.
3. The p-value trap
The KS statistic itself (max CDF gap, D) is fine. The problem is
thresholding on p < 0.05: the p-value shrinks with sample size, so
with millions of rows any real-world difference — including
harmless day-of-week variation — becomes "significant" and the alert
fires every window. Alert on the effect size (D above a
calibrated threshold, or PSI/JS bands) instead, or subsample to a
fixed n before testing. Rigor in monitoring means calibrated,
actionable thresholds, not small p-values.
Share this question