Advanced
Open
Pro
Evaluate the Uncertainty of a SKU-Level Demand Model
A supply-chain team ships 80 percent prediction intervals for weekly SKU demand and evaluates them on 10,000 held-out SKU-weeks. The results by segment:
| Segment | SKU-weeks | Observed coverage | Mean interval width |
|---|---|---|---|
| Fast movers | 6,000 | 0.84 | 20.0 |
| Slow movers | 3,500 | 0.74 | 6.0 |
| New SKUs | 500 | 0.52 | 9.0 |
- Compute the overall coverage. Does the headline look fine?
- Decide for each segment whether the shortfall is statistically meaningful. For new SKUs (0.52 on 500 points) give a 95 percent interval for the true coverage.
- Which metrics would you add beyond coverage and width, for the intervals and for a model that outputs a full predictive distribution? Explain the pinball-CRPS relation and why a point-forecast metric is not enough.
- Propose a fix that targets the failure and a monitoring plan. What can and cannot a conformal wrapper do here?
Share this question