Choose a fairness metric for a hiring screen and for a medical triage model
Two teams ask you which fairness metric to use.
- Hiring screen. A model ranks resumes and the top 30 percent go to a recruiter. The applicant pool has different group shares per role.
- Emergency triage. A model flags patients who need urgent attention among those arriving at an emergency department; a missed flag can be fatal, and a false flag costs a nurse's time.
For each: pick a primary criterion among demographic parity, equal opportunity, equalized odds, predictive parity and calibration within groups; name the check you would add; say what you would do about the label (what the model is trained to predict); and state what you give up. Then explain why you cannot simply demand all of them.
Hiring screen.
- Primary criterion: equal opportunity (equal TPR): among candidates who would succeed, each group should have the same chance of reaching the recruiter (Hardt et al. 2016; from memory). The harm is a false negative, a qualified person filtered out.
- Add the four-fifths check on selection rates, because employment-selection guidance in the US treats a ratio below 0.8 as evidence of adverse impact (from memory; a screen, not a legal test), and local rules may require published audits (check with counsel).
- Label: "hired" or "rated highly by past managers" encodes historic choices; prefer a measure of on-the-job outcomes or a structured assessment, and test for proxies such as names, schools and employment gaps.
- Give up: demographic parity (when base rates of qualification differ across applicant pools, equal opportunity will not give equal selection rates) and perfect predictive parity.
Triage.
- Primary criterion: equal false-negative rate across groups (equal opportunity on the "needs urgent care" class), together with calibration within groups so the clinician's reading of a risk score of, say, 0.6 means the same thing for every patient.
- Add: performance on small groups with intervals, and an evaluation by subgroup of the missed-case severity, not just counts.
- Label: the key trap is a proxy outcome. A health-system study found that predicting cost instead of need gave Black patients lower scores at equal illness, because less had been spent on their care (Obermeyer et al. 2019; from memory). Label with clinical outcome (admission, ICU transfer, deterioration within a window), not spend or utilization.
- Give up: equal false-positive rates, and possibly some overall efficiency. Because the FP cost is low, accept a higher FPR if it buys a lower FNR for the group with the highest miss rate.
Why not all of them. If the groups' base rates differ and the model is imperfect, calibration within groups, equal FPR and equal FNR cannot all hold (Chouldechova 2017; Kleinberg, Mullainathan and Raghavan 2016; from memory). In this lesson's credit simulation, a score calibrated within each group (mean predicted and observed repayment within 0.04 in every score band) with PPV 0.850 against 0.841 still approved the groups at 0.615 and 0.270, with TPRs of 0.688 and 0.364; forcing equal TPR then moved PPV to 0.850 against 0.751 (verified). So the answer is a choice, justified by who bears each error, signed off with legal and domain owners, documented in the model card, and re-audited.
A good closing sentence. "I would pick the criterion that equalizes the error that does the most harm, verify it with intervals on the groups that matter, check the label for proxy bias before touching the model, and write down what I traded away."
Share this question