Practice — Uncertainty Quantification & Conformal Prediction (6 questions)

A food-delivery team has a gradient-boosted model that predicts delivery time in minutes. They held out 400 labeled deliveries that the model never saw during training or tuning, and they want a 90 percent prediction interval to show customers ("arrives between X and Y").

1. Write a function that takes the fitted model, the 400 calibration points and new features, and returns the lower and upper bound using split conformal prediction. Justify every index you use.
2. For these 400 points, which order statistic of the absolute residuals do you take for \alpha=0.1, and for \alpha=0.05? If the relevant residual is 12.4 minutes and the model predicts 30 minutes, what is the interval?
3. A junior engineer proposes np.quantile(scores, 0.9) and, separately, calibrating on the training residuals "since there are more of them". Explain what each change does to the guarantee.
4. State precisely what the 90 percent guarantee says. A product manager asks: "so 90 percent of customers get an interval that contains their true time, and any one customer has a 90 percent chance?". Respond.

A retailer's demand model gets 90 percent split conformal intervals. They were calibrated on 500 residuals from February to April and measured 90.3 percent coverage on a May holdout. In November a large promotion calendar goes live; the model is unchanged. By mid-December the planner says "the intervals are lying: stockouts happened on items whose interval said it could not".

1. Define exchangeability and say precisely which part of the conformal argument fails here.
2. Give three distinct ways the retailer's data could violate it, and for each say whether the right fix is recalibration, weighted conformal, adaptive conformal inference, or a model change.
3. Quantify how bad it can get with a concrete example: if the residual noise doubles after a changepoint and the calibration is frozen, what long-run miscoverage should you expect, and what would online adaptation do?
4. What do you monitor so that this is detected in days, not weeks?