Sizing the Funnel from a Latency Budget
A video platform has a catalogue of 50 million videos. The ranking model costs 8 μs per (user, item) pair when batched, and the overall request budget (feature fetch + retrieval + ranking + re-ranking) is 120 ms p99.
- Show why scoring the full catalogue with the ranker is infeasible, with numbers.
- If feature fetch takes 15 ms and re-ranking takes 5 ms, how many candidates can the ranker afford to score within budget, and what does that number tell you about the candidate-generation stage's job?
- Candidate generation returns candidates from 4 sources (two-tower ANN, item-to-item, trending, subscriptions) with some overlap. Why is that better than relying on the two-tower source alone?
1. Full catalogue is infeasible
50,000,000 × 8 μs = 400,000,000 μs = 400 seconds per request, against a 120 ms budget — roughly a 3,300× overshoot. No batching or hardware improvement within a normal range closes a gap that size; the architecture must shrink the candidate set before ranking, not speed up the ranker.
2. Candidate count the ranker can afford
Remaining budget for ranking: 120 − 15 (features) − 5 (re-rank) = 100 ms = 100,000 μs. At 8 μs/item: 100,000 / 8 = 12,500 items, and in practice you'd leave margin for network and tail variance, so a real system might target 2,000–5,000. This tells you candidate generation's job is precise: it is not "return everything remotely relevant," it is "return at most a few thousand items with high recall of the items that would have ranked well," because anything beyond that count the ranker literally cannot afford to see.
3. Why multiple sources beat two-tower alone
Each retrieval source has different coverage and blind spots. The two-tower model is trained on historical interactions, so it is systematically weak on very recent items (not enough training signal yet) and can under-rank niche-but-relevant content it hasn't seen co-occur. Item-to-item captures "because you watched X" relationships even for items with sparse interaction counts. Trending surfaces what's happening right now, which the two-tower model — trained on a lagging window — cannot react to quickly. Subscriptions/follows guarantee that explicit user intent is never dropped even if the learned model happens to underweight it. Relying on one source means every blind spot of that source becomes a blind spot of the whole product; recall@k measured per source, then merged, is almost always higher than any single source's recall@k at the same total candidate count.
Share this question