Write an Evaluation Plan with GAUC, Time Splits and an Online Ladder

A new ranker beats the production model by 0.8 points of pooled AUC offline. Leadership wants to ship this week. The data science lead asks you for the evaluation plan. Facts: the user base has very different activity levels (a few users have hundreds of impressions per day and most have a few); the model is retrained daily; the old ranker's impressions are the only logged data; the new model adds sequence features computed from the user's history.

1. Explain why pooled AUC can mislead here and define GAUC. Two users: user A has 10 impressions and per-user AUC 0.90, user B has 100 impressions and AUC 0.60. Compute impression-weighted GAUC and the unweighted average. How should users with only one class be handled?
2. In the lesson's experiment three models with identical within-user ranking had pooled AUC 0.873, 0.665 and 0.257 and GAUC about 0.70 in each. What does this say about reading the 0.8-point gain?
3. Specify the offline split and feature rules, including how the sequence features are built, and the other offline metrics you would report.
4. Specify the online ladder (replay, interleaving, A/B, holdout) and the metrics and guardrails at each rung.
5. Name two reasons the offline gain might not reproduce online.