Explain Exchangeability and Diagnose a Violation After a Promotion Launch
A retailer's demand model gets 90 percent split conformal intervals. They were calibrated on 500 residuals from February to April and measured 90.3 percent coverage on a May holdout. In November a large promotion calendar goes live; the model is unchanged. By mid-December the planner says "the intervals are lying: stockouts happened on items whose interval said it could not".
- Define exchangeability and say precisely which part of the conformal argument fails here.
- Give three distinct ways the retailer's data could violate it, and for each say whether the right fix is recalibration, weighted conformal, adaptive conformal inference, or a model change.
- Quantify how bad it can get with a concrete example: if the residual noise doubles after a changepoint and the calibration is frozen, what long-run miscoverage should you expect, and what would online adaptation do?
- What do you monitor so that this is detected in days, not weeks?
1. Definition and failure. A sequence is exchangeable if its joint distribution is unchanged by any permutation; i.i.d. data are exchangeable. The conformal proof says the test score is equally likely to be any of the n+1 ranks among calibration plus test scores. That is only true if the test point is a permutation-symmetric partner of the calibration points. After the promotion the residuals have a different distribution from the February to April residuals, so the test score is systematically larger than a random calibration score and the rank is no longer uniform. The finite-sample guarantee is silently void; the code still runs and returns intervals.
2. Three violations and fixes.
- Covariate shift: promoted SKUs and promo-weeks are now a much larger share of traffic than in the calibration window, but given the features (including a promo flag) the demand relationship is unchanged. P(X) changed, P(Y\mid X) did not. Weighted conformal reweights calibration residuals by w(x)=p_{\text{new}}(x)/p_{\text{cal}}(x) (estimated with a domain classifier); in the lesson simulation plain conformal covered 0.842 on a shifted population and weighted conformal restored 0.902 by widening intervals (verified).
- Concept drift or a variance change: the relationship itself changed (promo uplift is larger than anything in training, shoppers stockpile). Weighted conformal does not help, since P(Y\mid X) moved. Use adaptive conformal inference to keep long-run coverage, and retrain or add promo features because ACI only widens intervals.
- Autocorrelation and non-stationarity in time: weekly demand residuals are serially correlated and seasonal, so a calibration set from one season is not exchangeable with another. Calibrate on a rolling, recent window and by horizon, and always calibrate on out-of-time residuals. Recalibration alone fixes this one if the process is otherwise stable.
3. How bad. In the lesson simulation the noise scale doubled at step 2,000 of 6,000 with the calibration frozen: static conformal's miscoverage rose from 0.10 to about 0.32 and stayed there (overall 0.2445), i.e. intervals covering roughly 68 percent when promising 90. ACI with \gamma=0.005 recovered to 0.100 within about 500 steps after a 0.135 miscoverage spike and ended with overall miscoverage 0.1030; with \gamma=0.02 the first post-shift window was 0.108 and the overall 0.1007 (verified; averages of 40 runs). The update is \alpha_{t+1}=\alpha_t+\gamma(\alpha-\text{err}_t). The trade-off is jumpier interval widths at large \gamma.
4. Monitoring. Track realized coverage online as soon as outcomes
arrive, in rolling windows, overall and by segment (promoted versus
non-promoted, category, store cluster); alert when a window falls outside
the Beta/binomial band for its size (at 500 points the sd of coverage is
about 0.013). Track the residual distribution (median and 90th percentile
of absolute residual) and input drift (PSI or a domain classifier AUC),
as in ml-monitoring-and-drift, plus interval width, which should rise
when ACI is active.
What a strong answer adds. The May holdout passing at 90.3 percent is not evidence of robustness: it came from the same regime. A planner's complaint is also a conditional-coverage complaint (the promoted items), which a marginal guarantee never promised. Say that you would report coverage separately for promoted items before the launch.
Share this question