Diagnosing an Evaluation Metric Mismatch
A candidate designs a fraud model and reports "ROC-AUC 0.98 on a random 80/20 split, so we are ready to ship." The fraud rate is 0.1 % and the manual review team can inspect 500 transactions per day out of 2 M.
- Identify at least three problems with this evaluation.
- Propose the evaluation you would present instead, including the split strategy and the metric tied to the review capacity.
- Explain how you would validate the model online without exposing the business to unbounded fraud losses.
1. Problems
- ROC-AUC on 0.1 % positives is misleading: the false-positive rate denominator is huge, so a model can have AUC 0.98 and still flood the review queue with false alarms. Precision–recall is the right view.
- Random split leaks time: fraud patterns evolve; a random split lets the model see future patterns while training and overstates performance. Duplicate or near-duplicate transactions from the same fraud ring can also land on both sides of the split.
- Label maturity ignored: chargebacks arrive weeks later, so recent rows are labelled "not fraud" simply because the label has not arrived. Training and evaluating on immature labels biases the model toward missing recent fraud.
- No baseline: 0.98 means nothing without the current rule engine's number on the same data.
- No operating point: the business decision is a threshold, and no threshold was reported.
2. Better evaluation
Time-based split (train on months 1–9, validate on month 10, test on month 11) with a label-maturity gap so every evaluated row has had at least 60–90 days to receive a chargeback. Report PR-AUC, and — because review capacity is 500/day out of 2 M (0.025 %) — precision and recall at the top-500 threshold, plus fraud dollars caught rather than counts, sliced by merchant category and country. Compare against the existing rules on identical rows.
3. Safe online validation
Shadow-score first (log predictions, take no action) to check distribution and latency. Then A/B on a small traffic slice with guardrails: cap the daily block volume, monitor customer-complaint and false-decline rates, and use the delayed chargeback signal as the primary metric knowing it matures over weeks. Keep the rules engine as a fallback so a bad model version can be rolled back instantly. Note that fraudsters adapt, so post-launch monitoring must watch for precision decay by segment.
Share this question