Forecasting at Scale: Global, Probabilistic & Hierarchical Models

The boundary: time-series-fundamentals covers decomposition, stationarity, ARIMA and ETS on a single series, single-series gradient-boosting features, basic backtesting and prediction intervals. This subject starts where it ends and does not repeat it: it is about what changes when there are thousands to millions of series, when the decision needs a quantile and not a mean, when many series are mostly zeros, and when the numbers must add up across a hierarchy. Interval theory that applies to any predictor (split conformal, coverage guarantees, adaptive intervals) is uncertainty-quantification-and-conformal-prediction; here we only use pinball loss and coverage to score forecasts. Time-to-event modelling for lifetimes is survival-analysis-and-time-to-event, metric and LTV definitions are product-analytics-metrics-and-ltv, the general monitoring playbook is ml-monitoring-and-drift, and generic metric theory is model-evaluation-metrics. Pricing and elasticity are covered in Pricing Optimization & Marketplace Economics; an end-to-end retail forecasting case study is forthcoming and is mentioned by title only.

Interviews ask this as "design the demand forecasting system for 2 million SKU-store pairs", "why one global model and not one ARIMA per series?", "how do you forecast a new product with no history?", "what is wrong with MAPE on slow movers?", "how do you make store forecasts add up to the region and the company?", "which loss do you train when the planner needs the 90th percentile?" and "how do you backtest this without leaking?". The senior answer names the structure (shared seasonality, hierarchy, intermittency, known-future covariates), picks a global model with scaled targets, scores with scale-free metrics against a naive baseline, and treats coherence and quantiles as product requirements, not afterthoughts.

Everything executable below was run (Python 3.12, NumPy 2.5, SciPy 1.18, pandas 2.3, scikit-learn 1.9, LightGBM 4.7, statsforecast 2.1, fixed seeds) on simulated panels with known structure, or by hand on 7 series, and is marked "(verified)". Claims recalled from the literature without re-running (the M4 and M5 competitions, N-BEATS, DeepAR, TFT, PatchTST, TimesFM, Chronos, Moirai, MinT details) are marked "(from memory)". I did not train a neural forecaster or run a foundation model; those sections are descriptive. The simulated panel: daily counts with a shared weekly profile and a shared yearly cycle, per-series scale spread over two orders of magnitude (5th to 95th percentile 2.2 to 135 units per day), promotions on about 6 percent of days with series-specific lift of 1.3 to 2.0, a slowly drifting level, AR(1) demand shocks (\phi=0.85), negative-binomial noise, and 30 percent growing series. Simulated numbers show mechanics and relative sizes, not benchmarks of a real retailer.