Beginner Open Free

Fraud Model Alerts and the Base Rate

A payments team deploys a fraud model. Historically, 0.2% of transactions are fraudulent. The model flags 95% of fraudulent transactions (recall / sensitivity) and incorrectly flags 2% of legitimate transactions (false-positive rate).

A transaction is flagged. What is the probability it is actually fraudulent? Show the calculation.
The team is disappointed and asks whether improving recall from 95% to 99% would fix the problem. Would it? What lever actually matters here?
The same model is deployed on a merchant segment where 5% of transactions are fraudulent. What is the precision of a flag now?

Solution

1. Posterior probability of fraud given a flag

Let F = fraud, + = flagged. P(F) = 0.002, P(+ \mid F) = 0.95, P(+ \mid F^c) = 0.02.

P(F \mid +) = \frac{0.95 \times 0.002}{0.95 \times 0.002 + 0.02 \times 0.998} = \frac{0.0019}{0.0019 + 0.01996} \approx 0.087

Only about 8.7% of flagged transactions are fraudulent. In natural frequencies: per 100,000 transactions, 200 are fraud and 190 of those are flagged; 99,800 are legitimate and 1,996 of those are flagged. Flags are dominated by false positives (1,996 vs 190) purely because legitimate transactions outnumber fraudulent ones 499 to 1.

2. Would better recall help?

Barely. With recall 0.99 the numerator becomes 0.99 \times 0.002 = 0.00198 and the posterior is 0.00198 / (0.00198 + 0.01996) \approx 0.090 — 9.0% instead of 8.7%. Recall is already high; the flood of false positives comes from the 2% false-positive rate applied to a huge population of legitimate transactions. The lever that matters is specificity: cutting the false-positive rate from 2% to 0.5% gives 0.0019 / (0.0019 + 0.00499) \approx 0.28 — more than tripling precision. When the base rate is tiny, the false-positive rate dominates the posterior.

3. Higher-prevalence segment

P(F) = 0.05:

P(F \mid +) = \frac{0.95 \times 0.05}{0.95 \times 0.05 + 0.02 \times 0.95} = \frac{0.0475}{0.0475 + 0.019} \approx 0.71

Precision jumps to about 71% with the identical model. Nothing about the model changed — the prior did. This is why a single "model accuracy" number is meaningless without the base rate of the population it will be applied to, and why the same classifier can be excellent in one segment and useless in another.

Share this question

Share on X Share on LinkedIn

← Back to Probability Fundamentals practice