Z-Score vs IQR vs MAD for Outlier Detection
A colleague flags outliers in a transaction-amount column using |z| > 3 and reports "no outliers found". You look at the data:
8, 9, 10, 10, 11, 12, 12, 13, 14, 15, 15, 16, 18, 20, 350
- Compute the z-score of the 350 value (use the sample standard deviation) and explain why the colleague's rule missed it.
- Apply the IQR rule (use Q_1 = 10, Q_3 = 15.5) and the MAD-based robust z-score. Do they flag 350?
- The row turns out to be a genuine bulk order from a business customer. Describe two different analyses in which you would handle this row differently, and what "handling" would mean in each.
1. Z-score and the masking effect
Sum = 533, n = 15, mean \approx 35.5. The deviations of the fourteen "normal" values from 35.5 are all in the −20 to −28 range, and the deviation of 350 is +314.5. Sum of squared deviations \approx 8{,}400 (from the fourteen small values) + 98{,}900 (from 350) \approx 107{,}300; sample variance \approx 107{,}300 / 14 \approx 7{,}660; s \approx 87.5.
z_{350} = (350 - 35.5)/87.5 \approx 3.6.
With these numbers 350 does clear 3 — but only just, and note what has happened: the mean has been dragged from ~13 to 35.5 and the SD from ~3.5 to 87.5 by the very point being tested. With a slightly smaller extreme value, say 250, z drops below 3 and the rule reports nothing. This is masking: the outlier inflates the scale it is measured against. Colleagues who report "no outliers by z-score" on right-skewed money data are usually reporting the failure of the method, not the cleanliness of the data. (Their claim of "no outliers" here also suggests they used the population SD or a stricter cutoff — either way, the method is fragile precisely where it is needed.)
2. Robust rules
IQR = 15.5 - 10 = 5.5. Upper fence = 15.5 + 1.5 \times 5.5 = 23.75. 350 is far above the fence — flagged. The quartiles barely move when one extreme value is added, so the fence is not corrupted.
MAD: median of the data is 13. Absolute deviations from 13: {5,4,3,3,2,1,1,0,1,2,2,3,5,7,337}; sorted, the median is 3. Robust z = (350 - 13)/(1.4826 \times 3) \approx 76. Flagged, and the magnitude is honest about how extreme it is.
3. Same row, different treatments
- Forecasting total daily revenue. The bulk order is real revenue. Keep it. If the model is sensitive to extremes, consider modelling business and consumer orders as separate streams, or winsorising at a high percentile for the consumer stream only — but the total must still add up to what finance sees.
- Estimating the "typical basket size" for a UX decision about the
checkout page. This question is about consumers; a business bulk
order is a different population. Exclude it by rule (e.g.
customer_type != "business"), not by value, document the exclusion, and report the median and IQR of the remaining data.
In both cases the action follows from investigating the row and stating the question — never from a threshold alone.
Share this question