Choosing the Ranking Objective
You are designing the homepage ranker for a video platform. A colleague proposes optimising purely for click-through rate (CTR), arguing "clicks are abundant, easy to log, and directly measure whether a user wanted to watch the video."
- Explain the specific failure mode this objective produces in production.
- Propose a better objective and justify each signal you include.
- Describe how you would validate that your chosen objective actually serves the business, given that offline metrics cannot fully verify it.
1. Failure mode of pure CTR optimisation
A CTR-only model learns that clickbait works: sensational thumbnails and titles that create curiosity gaps maximise the click probability regardless of whether the video satisfies the user. Users click, watch a few seconds, and abandon — CTR looks great while watch time, retention and trust erode. The model is optimising a proxy that is easy to game.
2. Better objective
Use a multi-objective score combining P(\text{click}) \times \mathbb{E}[\text{watch time} \mid \text{click}] as the workhorse (watch time is a much harder signal to game with thumbnails alone), plus P(\text{like}) and P(\text{survey satisfied}) as positive corrections, minus P(\text{dislike or not interested}) as an explicit penalty. Each signal patches a specific failure of the others: watch time alone still favours long videos and can reward "rabbit hole" content; likes and surveys are closer to true satisfaction but sparse; the negative signals catch clickbait and low-quality content that watch time might still reward if the user watches out of frustration or confusion.
3. Validating the objective
Offline metrics (AUC, NDCG using watch time as gain) can only confirm the model predicts its training labels well — they cannot confirm the labels are the right thing to optimise. Validate online: A/B test with guardrail metrics (skip rate, dislike rate, survey satisfaction, complaint rate) alongside the primary metric, and run a long-term holdback population that stays on the previous system for months to see whether short-term engagement gains translate into retention gains or instead into engagement followed by churn. If satisfied watch time rises but D7 retention or survey satisfaction falls, the objective weights need revisiting.
Share this question