Design an Abstention Policy for Automated Claims Triage

An insurer auto-decides 100,000 claims per day. A wrong automated decision costs $20 on average; sending a claim to a human reviewer costs $2 and (assume) the reviewer is right. The model's overall error rate is 9 percent. Sorting claims by an uncertainty score and splitting into ten equal groups, the error rate in each decile (from most certain to most uncertain) is 2, 2, 3, 3, 4, 5, 7, 10, 18 and 36 percent.

1. What is the daily cost of fully automating, and what is the break-even error rate for reviewing a claim?
2. Compute the expected daily cost if you review the top 1, 2, 3 and 4 most-uncertain deciles. Which is optimal, and why do two choices tie?
3. The review team can handle only 15,000 claims per day. What is the best policy and its cost?
4. Which uncertainty score would you use for a classifier and for a regression model, and how would you validate and monitor the policy? What does the conformal coverage guarantee say about the claims the model keeps?