Choose between direct, recursive and single-model horizon strategies for a 28-day forecast
You have to produce 28-day-ahead daily forecasts for 150,000 series with LightGBM. The nightly batch has 3 hours on a 32-core machine. A teammate proposes a recursive one-step model; another proposes 28 direct models; a third proposes one model with every lag at least 28 days back.
- Describe what each strategy trains on and what it feeds into the features at test time.
- Give the main failure mode of each.
- Which would you try first, and what experiment decides? What would change your mind?
- How do you report horizon-wise accuracy honestly, and what pitfall is hiding in a plot of error against horizon?
1. Mechanics.
- Recursive: one model trained to predict day t+1 from data up to t. To forecast 28 days, predict day 1, append the prediction to the history, compute the lags and rolling features again, predict day 2, and so on. At test time the features for later days are built from the model's own outputs.
- Direct: a separate model for each horizon h (28 models), each trained to predict day t+h from data up to t with lags at least h. Features are always built from real observations.
- Single window model: one model whose features are all at least 28 days old (lags 28, 29, 35, 42, ... and rolling means ending 28 days back) and which receives the weekday and promo of the target day as known-future features. It predicts any day in the window with the same function.
2. Failure modes. Recursive: errors compound, and the model never saw its own noisy outputs during training, so it can drift; it needs one full pass per day and the feature recomputation is serial. Direct: 28 models to train, store and monitor; separately trained horizons can give a jagged forecast curve; the cost grows with horizon. Window: it ignores the freshest 27 days of information, so it is weakest at short horizons.
3. Experiment. Run all three on the same rolling origins (at least 5, covering a peak) with the same features, and compare MASE and WAPE by horizon at the same horizon set. In the simulation (verified; 600 series, three seeds, horizon 14) the strategies finished within 1.3% of each other: MASE averaged over horizons was 0.888 (window), 0.877 (direct) and 0.881 (recursive); at h=1 the window model was 0.833 against 0.793 for direct and recursive; at h=14 recursive (0.828) began to trail direct (0.821). Seasonal naive was 1.211. So with moderately persistent demand the choice is mainly operational. Start with the single window model: one training job, one scoring pass, cheapest, easiest to monitor, and probably within a couple of percent. Add direct models for the short horizons only (for example h=1..7 direct plus one window model for the rest) if the backtest shows the loss at short horizons matters to the decision (production orders, staffing). Prefer direct over recursive in production because its inputs match training. Change my mind if demand is strongly autocorrelated (fast sellers, shocks that persist), where the freshest days are worth more and the window model loses at short horizons, or if the time budget permits 28 models easily.
4. Reporting. Report the metric by horizon (or horizon bands 1 to 7, 8 to 14, 15 to 28), pooled WAPE and per-series MASE, against seasonal naive. The pitfall: with a weekly pattern, the weekday of the target moves with the horizon, so a one-origin error-against-horizon plot mixes horizon with weekday (Mondays are easier or harder than Saturdays) and can look non-monotone. Average over many origins, or compare strategies at the same horizon, and do not read a trend from a single origin. Never compare a 1-step model on 1-step with another model on 14 steps.
Share this question