Design a Multi-Task Short-Video Ranker and Its Fusion Formula
A short-video app ranks 2,000 candidates per request. Product wants to optimize "satisfying watch time" and says: "just train on watch time, and also keep clicks and likes up". Logs show click and expected watch time are negatively correlated across candidates (about -0.4), likes are rare (about 3 percent of views) and "not interested" taps are rarer still (0.3 percent). An engineer proposes a shared-bottom network with five heads and a fusion formula V = p_{\text{click}} + 0.02\,E[\text{watch}] + 3\,p_{\text{like}} - 10\,p_{\text{dislike}}. A second engineer proposes MMoE "because it is state of the art".
- Pick the heads and the label definition for each (what is a positive, what is the regression target). Why is raw watch time a poor target?
- In the formula above, one candidate has p_{\text{click}} = 0.18, E[\text{watch}] = 60 s, p_{\text{like}} = 0.05 and p_{\text{dislike}} = 0.002; another has 0.30, 12 s, 0.02 and 0.001. Compute both values and say which term decides the ranking and why that is a scaling problem.
- A sweep of the watch weight from 0.001 to 0.005 on a candidate set moved the mean click probability of the top-20 slate from 0.653 to 0.079. What does that imply for how you set and ship the weights?
- When would you move from shared-bottom to MMoE or PLE? What do you measure to decide, and what does the lesson's toy experiment say about adopting it by default?
- Name two ways to make the fusion robust to the different scales of the heads.
1. Heads and labels. Use a completion/engagement head (positive when watch ratio exceeds a threshold, for example 70 percent of duration or 30 s, so a 2-second bounce is not a positive), a regression or bucketed head for expected watch ratio (target = watched seconds divided by duration, not raw seconds), a like head, a share or follow head, and a negative head for "not interested" or quick skip. Raw watch time is a poor target because it is dominated by video length (long videos win without being better), heavy-tailed, and pushes the feed toward long content; ratio or duration-bucketed targets remove most of that.
2. The arithmetic. Candidate 1: 0.18 + 0.02\cdot60 + 3\cdot0.05 - 10\cdot0.002 = 0.18 + 1.20 + 0.15 - 0.02 = 1.51. Candidate 2: 0.30 + 0.02\cdot12 + 3\cdot0.02 - 10\cdot0.001 = 0.30 + 0.24 + 0.06 - 0.01 = 0.59. The watch term (1.20 of 1.51) decides it. The reason is scale: seconds are about two orders of magnitude larger than probabilities, so a weight that "looks small" (0.02) dominates. The weight has no meaning until the heads are normalized.
3. Setting and shipping weights. In the lesson's synthetic sweep, mean click probability of the slate was 0.653, 0.624, 0.490, 0.205 and 0.079 at weights 0.001, 0.002, 0.003, 0.004 and 0.005, while mean watch time went from 5.5 s to 188 s. The response to a weight is steep and non-linear. So: normalize heads first (calibrated probabilities, watch ratio, or percentile ranks), change weights in small steps, tune online with guardrails (dislike and report rates, session length, next-day return), and do not set weights from intuition. Pareto reasoning also helps: only a few candidates are non-dominated on (click, watch), so a linear formula only reaches the convex part of that frontier; add constraints such as a minimum share of slots with high completion.
4. MMoE or PLE. Start with a shared bottom of adequate capacity and measure per-head metrics and gradient conflict. Move to MMoE when improving one head's metric degrades another's (negative transfer), or when tasks have clearly different input needs; move to PLE when a single mixture still shows the seesaw and you want task-specific plus shared experts. Log the average gate distribution per task to detect collapse onto one expert. In the lesson's toy experiment (two synthetic tasks with direction cosines 0.9, 0.5 and 0.0; parameter-matched models; three seeds) a plain shared bottom matched or beat MMoE in every setting (summed test MSE 0.061 vs 0.079 at cosine 0.9). The papers report gains on much larger data; the practical conclusion is "equal-capacity shared-bottom baseline first, then show the gain on your data".
5. Robust fusion. (a) Fuse percentile ranks within the candidate set (the lesson's rank fusion moved mean click smoothly through 0.482, 0.328 and 0.222 as the watch weight went 0.2, 0.5, 0.8), or fuse log-probabilities with exponents as weights; (b) standardize each head with a running mean and standard deviation so weights are in comparable units; (c) learn the weights from a long-term outcome or a survey-labeled satisfaction signal rather than from the training heads themselves.
Share this question