Advanced
Open
Pro
Diagnosing Calibration From a Reliability Table
A credit-default model with ROC-AUC 0.81 was trained with class_weight="balanced"
(default rate 4%). Its predicted probabilities on a 5,000-loan validation
set bin as follows:
| Bin | Loans | Mean predicted | Observed default rate |
|---|---|---|---|
| [0.0, 0.2) | 2,000 | 0.10 | 0.01 |
| [0.2, 0.4) | 1,500 | 0.30 | 0.03 |
| [0.4, 0.6) | 1,000 | 0.50 | 0.06 |
| [0.6, 1.0] | 500 | 0.75 | 0.14 |
- Compute ECE and describe the calibration failure. Why did it happen?
- The probabilities feed an expected-loss calculation. What goes wrong downstream, and does the AUC of 0.81 help?
- Propose a fix and say what data you would fit it on. Would fixing calibration change the ROC-AUC?
Share this question