Why Calibration Matters More Than AUC in an Ad Auction
Your team ships a new pCTR model. Offline AUC goes from 0.812 to 0.826 and log loss improves slightly. After launch, the finance team reports that video-ad revenue rose 18 % while image-ad revenue fell 11 %, and several large image advertisers complain their cost per click doubled.
A per-segment check shows the new model's sum(pCTR) / sum(clicks) is
1.24 for video ads and 0.85 for image ads (the old model was ~1.0 for
both).
- Explain, using the eCPM formula, why the auction outcome changed even though ranking quality improved.
- Two candidates cost the same bid ($2.00 CPC). The model predicts pCTR = 0.020 for a video ad and 0.017 for an image ad. What are their eCPMs? Using the calibration ratios above, what are the true expected CTRs, and which ad should have won?
- What would you add to the system so this cannot happen again, and how often would you refresh it?
1. Why the auction changed:
Ads are ranked and priced by eCPM = bid × pCTR × 1000. AUC only
measures whether positives are ranked above negatives overall; it says
nothing about the absolute probability scale within a segment. The new
model inflates pCTR by ~24 % on video and deflates it ~15 % on image, so
video ads win auctions they should lose and image ads lose auctions they
should win — and because second-price charging divides by the winner's
own pCTR, image winners are charged more per click. Discrimination
improved, calibration broke, and the marketplace mis-priced.
2. Worked numbers:
- Video: eCPM = 2.00 × 0.020 × 1000 = $40.0
- Image: eCPM = 2.00 × 0.017 × 1000 = $34.0 → video wins under the model.
Corrected CTRs (divide by the calibration ratio):
- Video true CTR ≈ 0.020 / 1.24 ≈ 0.0161
- Image true CTR ≈ 0.017 / 0.85 ≈ 0.0200
True eCPMs: video ≈ $32.3, image ≈ $40.0. The image ad should have won. The model reversed the correct outcome purely through miscalibration.
3. Fix and cadence: Add a post-hoc calibration layer fit per traffic segment (surface × country × ad format at minimum): Platt scaling where data per bucket is thin, isotonic regression where it is plentiful. Fit it on recent held-out labelled traffic and refresh hourly — it is a tiny model, so frequent refits are cheap and they track traffic-mix shifts the main model cannot. Add a launch gate that blocks any model whose per-segment calibration deviates more than ±5 % from 1.0, and a production alert on rolling predicted-vs-realised CTR by segment. Also check whether video clicks arrive with longer delay than image clicks — delayed-feedback bias can create exactly this segment-specific skew.
Share this question