Deriving Allow, Review and Remove Thresholds from Costs
A moderation team has a calibrated harm probability p for each post in a category where a missed harmful post costs M=120 cost units, removing a benign post costs F=15, one human review costs R=1, and the human reviewer is wrong 5 percent of the time (e=0.05, in either direction). The units are placeholders agreed with the policy owner, not dollars. Prevalence is about 0.4 percent.
The current system removes everything above p=0.5 and allows the rest; there is no review lane. The reviewer team has spare capacity.
- Write the expected cost of each of the three actions, then derive the two thresholds. Evaluate them for the numbers above.
- For posts at p=0.01, p=0.20 and p=0.70, which action is cheapest and by how much?
- Reviewer capacity drops by half. Show, with a shadow price, how the thresholds move and what you give up.
- The policy owner doubles the missed-harm cost to 240. What changes, and which threshold moves more?
1. Expected costs and thresholds. For a post with probability p:
The review cost has three parts: the reviewer's fee R; with probability e a harmful post is wrongly cleared (cost eM weighted by p); and with probability e a benign post is wrongly removed (cost eF weighted by 1-p). Setting c_{\text{allow}}=c_{\text{review}} and c_{\text{review}}=c_{\text{remove}} gives
So: allow below 1.5 percent, review between 1.5 and 65.4 percent, remove above 65.4 percent. The existing single cut at 0.5 sits in the middle of the review band: it removes posts the cost matrix wants a human to look at, and it allows posts at p=0.4 that cost on average 0.4 x 120 = 48 units against a review cost of about 3.85.
2. The three example posts.
| p | allow | review | remove | cheapest |
|---|---|---|---|---|
| 0.01 | 1.20 | 1.80 | 14.85 | allow |
| 0.20 | 24.00 | 2.80 | 12.00 | review (4.3x cheaper than removal) |
| 0.70 | 84.00 | 5.43 | 4.50 | remove (review costs 21 percent more) |
At p=0.01 the post is probably benign and reviewing it costs 1.80 against an expected missed-harm cost of 1.20. At 0.70 the post is more likely harmful than not and the automatic removal is cheaper than paying for a review whose own error rate is 5 percent.
3. Halved capacity and the shadow price. Treat the reviewer's time as scarce: add a per-review shadow price \lambda to R and recompute. Using the same formulas:
| \lambda | allow / review boundary | review / remove boundary |
|---|---|---|
| 0 | 0.0153 | 0.654 |
| 1 | 0.0240 | 0.605 |
| 2 | 0.0327 | 0.556 |
| 4 | 0.0501 | 0.457 |
Raising \lambda shrinks the review band from both sides: more uncertain-low posts are allowed and more uncertain-high posts are removed automatically. Choose \lambda so that the expected review volume (the share of posts with p in the band, from the calibrated score distribution) matches the halved capacity. The cost is explicit: in the case-study simulation going from 71,480 to 32,830 review items a day cost about 10 recall points. Better levers are a stronger score for the uncertain band (selective routing to a second tier) or a priority lane that spends scarce review time on the high-severity items.
4. Doubling the missed-harm cost. Only the allow/review boundary depends strongly on M: p_{ar}=1.75/(228+0.75)=0.0077, half its old value, because M sits in its denominator. The review/remove boundary uses M only through the small term eM: it moves from 0.654 to 13.25/(12+14.25)=0.505, a smaller shift in relative terms but meaningful because it also pushes more posts to automatic removal. The combined effect is more posts reviewed, so budget the extra reviewer load before changing the cost value, and document the owner of the number: thresholds are an auditable consequence of stated values.
Key points. (a) Calibrated probabilities are required; a model trained on an enriched sample needs a prior correction first. (b) The two thresholds have closed forms. (c) A single cut has no cheap action for the uncertain region. (d) Reviewer error e belongs in the model; ignoring it overstates the value of review. (e) Capacity enters through a shadow price, not by moving thresholds by hand.
Share this question