Conformal Prediction Sets for a 1000-Class Product Classifier

An e-commerce team has a product-categorization model with 1,000 leaf categories. Accuracy is 78 percent, class frequencies follow a heavy tail (a few categories have 5 percent of traffic each, hundreds have under 0.01 percent), and the model is overconfident. They want to show the top candidates to a merchandiser and promise "the right category is in the list at least 90 percent of the time". They have 100,000 labeled calibration examples.

1. Why is "show the top-5" or "show classes until probabilities sum to 0.9" not a guarantee? Describe the LAC, APS and RAPS conformal scores and pick one for this problem.
2. What does the conformal guarantee say here, and is it a per-category promise? Why can't you simply calibrate one threshold per category? Quantify with the calibration budget.
3. How would you get better coverage for rare categories without per-class thresholds?
4. What set size would you expect, how do you make the sets usable, and what would you monitor?