Reading a Confusion Matrix Under Imbalance
A defect-detection model is evaluated on 20,000 manufactured parts, of which 200 are defective (1% prevalence). At the default 0.5 threshold:
Pred defective Pred OK
Actual defective 120 80
Actual OK 280 19,520
- Compute accuracy, precision, recall, specificity, F1 and MCC. Compare accuracy to the "predict OK for everything" baseline.
- A stakeholder says "99% accuracy — ship it". Explain, using the numbers, why that statement is misleading in both directions (it overstates and understates the model).
- Which single metric would you put on the dashboard for this model and why?
1. Metrics:
- Accuracy = (120 + 19,520) / 20,000 = 0.982
- Baseline (all OK) accuracy = 19,800 / 20,000 = 0.990 — the model's accuracy is lower than the trivial constant.
- Precision = 120 / (120 + 280) = 0.30
- Recall = 120 / 200 = 0.60
- Specificity = 19,520 / 19,800 = 0.986; FPR = 0.014
- F1 = 2 × 0.30 × 0.60 / 0.90 = 0.40
- MCC = (120 × 19,520 − 280 × 80) / sqrt(400 × 200 × 19,800 × 19,600) = (2,342,400 − 22,400) / sqrt(3.104 × 10^13) ≈ 2,320,000 / 5,571,000 ≈ 0.42
2. Why "99% accuracy" misleads:
It overstates because a model that flags nothing scores 99.0% — accuracy is dominated by the 19,800 OK parts and says nothing about whether defects are caught. Any accuracy number near the base rate is compatible with a useless model.
It understates because this model actually catches 60% of defects while only bothering inspectors with a 1.4% false-alarm rate on good parts, and its accuracy (98.2%) is below the do-nothing baseline. Judged by accuracy you would reject a model that is clearly finding real signal (MCC 0.42, precision 30% against a 1% base rate — a 30× lift). Accuracy simply cannot rank "does nothing" against "does the job" when prevalence is 1%.
3. Dashboard metric:
For monitoring, pick something tied to the operational decision. Two defensible answers:
- Recall at a fixed precision (or at a fixed alert budget) — e.g. "recall at ≤ 400 alerts/day". This maps to what inspection can absorb and what escapes to customers.
- PR-AUC / average precision if the threshold is retuned regularly and you want a threshold-free view of the positive class.
MCC is a good single scalar for model comparison, but stakeholders understand "we catch 60% of defects at 300 false alarms per 20k parts" better than a correlation coefficient. Whatever you pick, always show the base rate next to it.
Share this question