Practice — Case Study: Content Moderation System (5 questions)

A moderation team has a calibrated harm probability p for each post in a category where a missed harmful post costs M=120 cost units, removing a benign post costs F=15, one human review costs R=1, and the human reviewer is wrong 5 percent of the time (e=0.05, in either direction). The units are placeholders agreed with the policy owner, not dollars. Prevalence is about 0.4 percent.

The current system removes everything above p=0.5 and allows the rest; there is no review lane. The reviewer team has spare capacity.

1. Write the expected cost of each of the three actions, then derive the two thresholds. Evaluate them for the numbers above.
2. For posts at p=0.01, p=0.20 and p=0.70, which action is cheapest and by how much?
3. Reviewer capacity drops by half. Show, with a shadow price, how the thresholds move and what you give up.
4. The policy owner doubles the missed-harm cost to 240. What changes, and which threshold moves more?

Your platform allows 4.9 million of its 5 million daily posts without human review. A user-report audit suggests missed harm exists but is biased toward popular posts. You can afford 5,000 expert labels per week. The model's posterior splits the allowed bucket into three strata with these (assumed) sizes and true harm rates:

The model actioned (removed or confirmed in review) an estimated 10,500 harmful posts a day.

1. How many harmful posts per day are missed, and what is the recall?
2. A simple random sample of 5,000 allowed posts: what is the relative standard error of the missed-rate estimate? What happens if it finds no positives?
3. Design the stratified sample (allocation) and compare its standard error with the simple random sample's.
4. How many labels are needed for plus or minus 25 percent and plus or minus 50 percent relative precision, and what would you tell the business?