Implement Split Conformal Intervals for a Delivery-Time Model
A food-delivery team has a gradient-boosted model that predicts delivery time in minutes. They held out 400 labeled deliveries that the model never saw during training or tuning, and they want a 90 percent prediction interval to show customers ("arrives between X and Y").
- Write a function that takes the fitted model, the 400 calibration points and new features, and returns the lower and upper bound using split conformal prediction. Justify every index you use.
- For these 400 points, which order statistic of the absolute residuals do you take for \alpha=0.1, and for \alpha=0.05? If the relevant residual is 12.4 minutes and the model predicts 30 minutes, what is the interval?
- A junior engineer proposes
np.quantile(scores, 0.9)and, separately, calibrating on the training residuals "since there are more of them". Explain what each change does to the guarantee. - State precisely what the 90 percent guarantee says. A product manager asks: "so 90 percent of customers get an interval that contains their true time, and any one customer has a 90 percent chance?". Respond.
1. The function.
import math, numpy as np
def split_conformal_interval(model, X_cal, y_cal, X_new, alpha=0.1):
scores = np.abs(y_cal - model.predict(X_cal)) # nonconformity scores
n = len(scores)
k = math.ceil((n + 1) * (1 - alpha)) # finite-sample index
q = np.inf if k > n else np.sort(scores)[k - 1] # k-th smallest (1-based)
pred = model.predict(X_new)
return pred - q, pred + q
The score is the absolute residual, so the set of labels consistent with
the calibration data is \hat\mu(x)\pm\hat q. The index uses n+1
because the new point's score is ranked among n+1 scores; under
exchangeability its rank is uniform on 1,\dots,n+1, so
P(s_{n+1}\le s_{(k)})\ge k/(n+1)\ge1-\alpha. The ceiling rounds up to be
conservative. If k\gt n the threshold is infinite (the interval is the
whole line) rather than an invented number. With a 0-based array the
k-th smallest is sorted[k-1]: the off-by-one is the most common bug.
2. The indices. \alpha=0.1: k=\lceil401\times0.9\rceil=\lceil360.9\rceil=361. \alpha=0.05: k=\lceil401\times0.95\rceil=\lceil380.95\rceil=381. With \hat q=12.4 and a prediction of 30 the interval is [17.6,\ 42.4] minutes (arithmetic; the indices were also checked in Python). With only 8 calibration points k=\lceil9\times0.9\rceil=9\gt8 and it returned (-\infty,\infty) (verified), which is correct behaviour: you cannot claim 90 percent from 8 points.
3. The two shortcuts.
np.quantile(scores, 0.9)interpolates at position 0.9(n-1), which is a smaller order statistic than \lceil(n+1)\times0.9\rceil, so the threshold is slightly too small and the guarantee \ge1-\alpha is lost (in a 500-point example the plain quantile was 2.7933 against the correct 2.8044; verified). The effect shrinks with n and is noticeable under about 100 points.- Training residuals are optimistic: the model was fit on those points, so they are not exchangeable with new points and the scores are too small. Coverage then falls below target (in the lesson's simulation 0.876 for a depth-3 GBM against the 0.90 target, and 0.879 for a deliberately overfit GBM against 0.905 with a held-out calibration set; verified). More points do not help; independence from the fit is what matters. The same applies to data used to tune hyperparameters or choose among models.
4. The guarantee, and the manager's question. For exchangeable data, 1-\alpha\le P(Y_{n+1}\in C(X_{n+1}))\le1-\alpha+1/(n+1), where the probability is over the draw of the calibration set and the new delivery. It is distribution-free and holds for any model. It does not say that this customer has a 90 percent chance, or that every kind of order is covered at 90 percent. Conditional on one fixed calibration set the coverage is Beta distributed, \text{Beta}(361,40) here, with mean 0.9002, standard deviation 0.0149 and a 5th to 95th percentile range of 0.875 to 0.924 (computed from the Beta law; verified), so a single deployment could sit at 88 percent without anything being wrong.
What a strong answer adds. Check that the interval widths are uniform: absolute-residual conformal gives the same width for a 2 km lunch order and a 12 km storm-night order, so it will over-cover the easy deliveries and under-cover the hard ones; mention normalized residuals or conformalized quantile regression as the fix and audit coverage by distance and time-of-day bucket. Mention scheduling a recalibration with fresh data, since delivery times drift.
Share this question