Practice — Case Study: Demand Forecasting for Retail & Logistics (5 questions)
Your Backtest Prefers the Model That Ignores Stockouts Permalink →
A retailer forecasts daily demand per SKU and store. Recorded sales are censored whenever the shelf empties. In a synthetic panel where the true demand is known, you train the same global Tweedie model four ways and score five rolling origins (a 28-day test window each):
| Training treatment | Bias vs true demand | WAPE vs true demand | WAPE vs recorded sales | Bias on promo days |
|---|---|---|---|---|
| A: raw sales | -6.9% | 0.5936 | 0.6110 | -18.3% |
| B: drop censored target rows | -4.9% | 0.5897 | 0.6154 | -10.8% |
| C: impute E[d given d at least s] for censored sales | -1.9% | 0.5923 | 0.6235 | -5.9% |
| Oracle (true mean) | +0.1% | 0.5730 | 0.6168 | -0.5% |
Overall, 7.3 percent of cells are stockouts, but they are 16.6 percent of promo-day cells.
- If the team chooses by WAPE against recorded sales, which treatment wins, and why is that the wrong decision?
- Treatment B has the best WAPE against true demand yet you do not recommend it. Why?
- How would you evaluate a censoring fix on real data, where true demand is never observed?
- What does the fix need as a prerequisite, and what do you monitor so it does not silently stop working?
Share this question
Which Quantile Do You Order For a 95 Percent Service Level? Permalink →
A planner asks for "a 95 percent service level" on every SKU. The order covers a 9-day protection interval (7 days between reviews plus a 2-day supplier lead time). You have a model for the 9-day demand sum, with these realised results on 19,600 held-out series-cycles (mean demand 18.1 units per cycle):
| Requested quantile | Cycle service level | Fill rate | Units left over per cycle |
|---|---|---|---|
| 0.80 | 78.7% | 95.0% | 4.66 |
| 0.90 | 87.7% | 97.2% | 6.57 |
| 0.95 | 92.5% | 98.2% | 8.38 |
| 0.98 | 96.0% | 98.9% | 10.59 |
A colleague proposes summing the nine daily 95th percentiles instead. That policy gives a 99.8 percent cycle service level and 24.51 units left over per cycle.
- Which request meets "95 percent service" if it means cycle service, and which if it means fill rate? What do the two cost in stock?
- Why does the colleague's sum over-deliver?
- For a thin-margin perishable item the cost of one unit short is 25 percent of price and the cost of one unit wasted is 75 percent. What quantile minimises expected cost? Is 95 percent sensible there?
- What is the cost ratio hidden behind a 95 percent target, and how would you fix the shortfall at the 0.95 request?
Share this question