Why PR-AUC Beats ROC-AUC at 0.1% Fraud Prevalence
Your fraud model is evaluated on 2,000,000 transactions containing 2,000 fraud cases (0.1 %). Model A flags 12,000 transactions and catches 1,600 fraud. Model B flags 5,000 transactions and catches 1,400 fraud.
- Compute the recall, false-positive rate, and precision for each model.
- Explain why the two models look almost identical on an ROC curve but very different on a precision-recall curve.
- Which model would you ship if manual review capacity is 5,000 transactions/day, and why?
1. Numbers: Negatives = 1,998,000 for both models.
Model A: recall = 1600/2000 = 80 %; FP = 12,000 − 1,600 = 10,400; FPR = 10,400/1,998,000 ≈ 0.52 %; precision = 1600/12,000 ≈ 13.3 %.
Model B: recall = 1400/2000 = 70 %; FP = 5,000 − 1,400 = 3,600; FPR = 3,600/1,998,000 ≈ 0.18 %; precision = 1400/5,000 = 28 %.
2. Why ROC hides the difference: ROC plots recall against FPR, and FPR is computed over the ~2,000,000 negatives, so even a large absolute change in false positives (10,400 vs 3,600 — a 3× difference) barely moves the FPR (0.52 % vs 0.18 %); both points sit near the origin and the curves look close. Precision divides by the flagged set size instead of the full negative population, so it is sensitive to exactly the quantity that determines review-team and customer-facing cost. That is why PR-AUC (or precision at a fixed operating point) is the metric that reflects what the business feels at extreme imbalance, while ROC-AUC can look reassuring for a materially worse model.
3. Which to ship: With a hard cap of 5,000 reviews/day, Model A's 12,000 flags are not operationally deployable without a second triage stage, and even routing only its top 5,000 by score would need re-evaluation at that cut. Model B fits the capacity directly: 5,000 flagged, 1,400 caught, 28 % precision — the review team clears the whole queue and 72 % of what they review is legitimate but flagged. Unless Model A's top 5,000 by score outperforms Model B head-to-head (must be checked explicitly, not assumed from the aggregate numbers), Model B is the practical choice given the stated constraint.
Share this question