Translating a Business Objective into an ML Objective
A product manager at a short-video app says: "We want people to spend more time in the app. Build a model for the For You feed."
- Propose a concrete ML objective (label, prediction unit, decision) that serves this goal, and name the gap between your proxy and the true business goal.
- Give one way the naive proxy could be gamed by content creators, and how you would adjust the objective or the re-ranking layer to defend against it.
- Which offline metric and which online metric would you report, and why are they different?
1. ML objective
- Prediction unit: a (user, video, context) triple at impression time.
- Label: several implicit signals, primarily P(\text{watch} \ge 30\text{s}) or expected watch fraction (seconds watched / video length, capped), plus secondary labels such as like, share and "not interested".
- Decision: rank the ~1,000 retrieved candidates by a weighted combination, e.g. s = w_1 P(\text{watch}\ge30\text{s}) + w_2 P(\text{like}) - w_3 P(\text{not interested}), and show the top ~20.
- Gap: "time in app" is a session-level, long-horizon quantity; the proxy is a per-item, short-horizon quantity. Optimising per-item watch can still shorten sessions if it fills the feed with repetitive or exhausting content, and it says nothing about retention next week.
2. Gaming and defence
Creators can inflate P(\text{watch}\ge30\text{s}) with misleading hooks or by making every video exactly 31 seconds so completion looks high. Defences: use watch fraction rather than a fixed threshold so short videos are not favoured; add negative labels (skip within 2 s, "not interested", reports) with meaningful weight; and in the re-ranker enforce creator/topic diversity and demote content whose engagement comes with high report rates. Mention that any single proxy will be Goodharted, so the objective is a portfolio of labels tuned by A/B.
3. Offline vs online metrics
Offline: NDCG@k or AUC per label on a time-based hold-out, sliced by new vs returning users — these measure whether the ranker orders items correctly given logged data. Online: A/B on total time in app per user per day and 7-day retention, with guardrails on report rate and latency. They differ because offline data is biased by what the old system chose to show and cannot capture session-level or long-horizon effects; offline gains are a necessary but not sufficient condition for shipping.
Share this question