Uncertainty Quantification & Conformal Prediction
The boundary: probability calibration and ECE for classifiers live in model-evaluation-metrics, forecast-interval specifics (horizon-dependent variance, ARIMA/ETS intervals, backtests) live in time-series-fundamentals and Forecasting at Scale: Global, Probabilistic & Hierarchical Models lesson, and posterior and credible-interval reasoning lives in bayesian-statistics-for-data-scientists. This subject covers the model-agnostic layer on top of any predictor: how to attach an interval or a set to a prediction, what can be guaranteed about it, what cannot, and what to do with the answer. Distribution-shift detection itself is in ml-monitoring-and-drift; here we only show how to keep intervals honest when the shift is known or tracked.
Interviews ask this as "how would you give a prediction interval for a gradient-boosted model?", "what does 90 percent coverage guarantee, and what does it not?", "explain conformal prediction in two minutes", "your model is calibrated, so why is the interval wrong?" and "how do you handle 1000 classes?". The senior answer names the guarantee (marginal, finite-sample, distribution-free, assuming exchangeability), shows the local failure (conditional coverage), and says what to monitor.
Everything executable below was run (Python 3.12, NumPy 2.5, SciPy 1.18, scikit-learn 1.9, fixed seeds) on simulated data with known truth and is marked "(verified)". Facts recalled from the literature without re-running are marked "(from memory)". The LLM-uncertainty paragraph is entirely from memory. Simulated numbers show mechanics and relative sizes, not benchmarks.
The regression simulation. One feature x\sim U(0,10), mean f(x)=2\sin x+0.5x, Gaussian noise with standard deviation \sigma(x)=0.3+0.25x, so the noise is 0.55 at x=1 and 2.30 at x=8. The model is a depth-3 HistGradientBoostingRegressor; 2,000 training points, 500 calibration points, 100,000 test points (as in the worked example). Because \sigma(x) varies, a single interval width is wrong almost everywhere, which is exactly what makes this a good testbed.