Your Backtest Prefers the Model That Ignores Stockouts
A retailer forecasts daily demand per SKU and store. Recorded sales are censored whenever the shelf empties. In a synthetic panel where the true demand is known, you train the same global Tweedie model four ways and score five rolling origins (a 28-day test window each):
| Training treatment | Bias vs true demand | WAPE vs true demand | WAPE vs recorded sales | Bias on promo days |
|---|---|---|---|---|
| A: raw sales | -6.9% | 0.5936 | 0.6110 | -18.3% |
| B: drop censored target rows | -4.9% | 0.5897 | 0.6154 | -10.8% |
| C: impute E[d given d at least s] for censored sales | -1.9% | 0.5923 | 0.6235 | -5.9% |
| Oracle (true mean) | +0.1% | 0.5730 | 0.6168 | -0.5% |
Overall, 7.3 percent of cells are stockouts, but they are 16.6 percent of promo-day cells.
- If the team chooses by WAPE against recorded sales, which treatment wins, and why is that the wrong decision?
- Treatment B has the best WAPE against true demand yet you do not recommend it. Why?
- How would you evaluate a censoring fix on real data, where true demand is never observed?
- What does the fix need as a prerequisite, and what do you monitor so it does not silently stop working?
1. The recorded-sales backtest rewards the wrong model. Against recorded sales, A scores 0.6110 and C scores 0.6235; even the oracle (0.6168) loses to A. Recorded sales are themselves censored, so a model that learned the censored pattern reproduces the data it is scored on. A forecaster tuned this way under-orders exactly where it hurts: on promo days A is 18.3 percent low against 5.9 percent for C. The decision metric should be built from demand, or from a stockout-aware slice.
2. B is a selection fix, not a correction. Dropping rows whose target day stocked out removes the high-demand days, so the model sees a distribution with its top cut off. It still under-forecasts by 4.9 percent overall and 10.8 percent on promos. Its small WAPE advantage (0.5897 against 0.5923) is within what five origins can resolve, whereas the bias gap is a systematic 3 points overall and 5 points on promos. Flag features (treatment D in the lesson, -5.2 percent) have the same issue: even with the flag set to zero at scoring time, the model has learned the relationship between features and sales on days that happened to stock out.
3. Evaluating without true demand. (a) Restrict scoring to uncensored cells and report bias separately on slices where stockouts concentrate (promos, fast sellers, new items); (b) run a small randomised or planned-overstock test in a few stores to observe uncensored demand; (c) check that total imputed units are plausible against shelf capacity and lead-time stock, since imputation recovered only about 74 percent of the 248,000 lost units in the training window of the lesson (4.43 million sold, 4.61 million imputed, 4.67 million true); (d) track the fraction of days with positive stock and zero sales, which should be explained by genuine zero demand. A censored-likelihood (Tobit style) loss is the principled alternative to imputation.
4. Prerequisite and monitoring. A trusted record of stock on hand at the start of each day: a stockout is stock reaching zero with demand at least the stock, not a day of zero sales. If the inventory feed drifts (phantom stock, late files) every flag-based method inherits the noise. Monitor the share of stockout-flagged cells per store and category, days with zero sales on positive stock, negative stock counts and stores with missing feeds. A sudden drop in the flagged share usually means a broken feed, which silently switches the correction off. Also keep bias, not only WAPE, on the dashboard: in this panel WAPE moved by 0.0013 while promo-day bias moved by 12 points.
Share this question