Measuring Recall When You Cannot See What You Missed
Your platform allows 4.9 million of its 5 million daily posts without human review. A user-report audit suggests missed harm exists but is biased toward popular posts. You can afford 5,000 expert labels per week. The model's posterior splits the allowed bucket into three strata with these (assumed) sizes and true harm rates:
| Stratum | Allowed posts per day | Harm rate |
|---|---|---|
| Low posterior | 3.0 million | 0.0001 |
| Middle | 1.5 million | 0.0008 |
| High | 0.4 million | 0.005 |
The model actioned (removed or confirmed in review) an estimated 10,500 harmful posts a day.
- How many harmful posts per day are missed, and what is the recall?
- A simple random sample of 5,000 allowed posts: what is the relative standard error of the missed-rate estimate? What happens if it finds no positives?
- Design the stratified sample (allocation) and compare its standard error with the simple random sample's.
- How many labels are needed for plus or minus 25 percent and plus or minus 50 percent relative precision, and what would you tell the business?
1. Misses and recall. Missed harm per day = 3.0M x 0.0001 + 1.5M x 0.0008 + 0.4M x 0.005 = 300 + 1,200 + 2,000 = 3,500. The pooled miss rate is 3,500 / 4.9M = 0.000714 (0.0714 percent). Recall = 10,500 / (10,500 + 3,500) = 0.75. The missed harm is concentrated: the high stratum holds 8 percent of the allowed posts and 57 percent of the misses.
2. Simple random sampling. With n=5{,}000 and rate r=0.000714, the expected number of positives is 3.6 and the standard deviation of the estimate is \sqrt{r(1-r)/n}=0.000378, a relative standard error of 53 percent. In about 3 percent of weeks the sample contains zero positives. With zero positives in n audited posts the 95 percent upper bound is roughly 3/n (the rule of three): 0.0006 (0.06 percent) for 5,000 posts, which, applied to 4.9M posts, still allows 2,940 missed posts a day. Zero hits in 3,000 posts gives an upper bound of 0.00100 and zero in 30,000 gives 0.00010. "We found nothing" bounds the rate; it does not demonstrate that recall is high.
3. Stratified design. Stratify by the model posterior and allocate by Neyman: n_h\propto N_h\sqrt{r_h(1-r_h)}. The shares are 29.8, 42.1 and 28.0 percent, so the 5,000 labels become about 1,491 / 2,107 / 1,402 (instead of 3,061 / 1,531 / 408 under proportional allocation). The estimator is \hat r=\sum_h (N_h/N)\hat r_h. Its standard deviation at n=5{,}000 is 0.000290 (relative error 41 percent) against 0.000378 (53 percent) for the simple sample, a variance ratio of 1.7. At 20,000 labels the relative errors are 20 percent and 26 percent. In practice add a per-stratum minimum (say 150) so a lucky or unlucky empty stratum cannot dominate, and use a bootstrap or a Bayesian interval, since the plain stratified variance is crude when strata hold only a handful of positives.
4. Labels required. For relative half-width \epsilon at 95 percent confidence, n=z^2(1-r)/(r\epsilon^2):
| Target | Simple random sample | Neyman stratified |
|---|---|---|
| plus or minus 50% | 21,498 | 12,700 |
| plus or minus 25% | 85,990 | 50,802 |
At 90 seconds a label, 12,700 labels is about 318 expert hours. The honest answer to the business is that a weekly 5,000-label budget gives a miss rate known to roughly plus or minus 80 percent (about 1.96 x 41 percent) and a recall interval of roughly 10{,}500/(10{,}500+3{,}500\cdot 1.41)=0.68 to 10{,}500/(10{,}500+3{,}500\cdot 0.59)=0.84 at one standard error, assuming the actioned count is known exactly. Pool four to six weeks for a tighter trend line, sample more heavily in the high stratum when severity is high, and use the audit as a monitor of large changes, not of small ones.
Key points. (a) Misses live in the allowed bucket, not in the flags or the reports. (b) A random sample of a rare class finds few positives. (c) Stratify by model score, allocate by Neyman, floor each stratum. (d) Quote intervals and sample-size arithmetic. (e) The audit sample must stay independent of the model and a fixed share of every training set so it remains an unbiased anchor.
Share this question