Ranking & Recommendation System Architecture

Almost every ML system design interview question at a consumer company is, underneath, the same question: given a user and a context, choose the best k items out of a catalogue of millions in under 100 ms, using data the system itself generated. Video feeds, product recommendations, ad selection, search ranking, "people you may know", notification selection — they all share one architecture, and interviewers expect you to draw it without prompting.

That architecture is a funnel: a cheap stage that narrows millions of items to a few thousand plausible candidates, an expensive model that scores those thousand precisely, and a final policy layer that reorders the top few for diversity, freshness and business rules. Around it sits the machinery of training data construction, debiasing, evaluation and experimentation. Getting the funnel right is the difference between an answer that sounds like a Kaggle notebook ("I'd train a model to predict click") and an answer that sounds like someone who has shipped a feed.

This subject teaches the general architecture. The Case Study: Video Recommendation and Case Study: Search Ranking subjects apply it end to end to a specific product with specific numbers; the ML Monitoring, Drift & Retraining subject covers what happens after launch.

Why a Funnel: The Compute Budget

Suppose a catalogue of N = 10^8 items and a ranking model that costs roughly 10 μs per (user, item) pair when batched on an accelerator. Scoring everything for a single request would take

against a latency budget of ~100 ms end-to-end. Even a model 100× cheaper is 10 s. No amount of hardware fixes a 10^4× gap. So the system is split into stages with a shrinking item count and a growing per-item cost:

Stage             Items in → out     Cost per item      Total (approx)   Model
───────────────   ────────────────   ────────────────   ──────────────   ────────────────────
Candidate gen     10^8  → ~2,000     ~0 (ANN index)     5–15 ms          two-tower / heuristics
Ranking           ~2,000 → ~2,000    ~10 μs (batched)   20–40 ms         deep multi-task model
Re-rank / policy  ~2,000 → k=20      ~10 μs (top-100)   1–5 ms           rules + light model
Feature fetch                                           10–20 ms         online feature store
                                                        ─────────
                                                        ≈ 60–80 ms p50, target < 100–150 ms p99

The principle: each stage should be as accurate as it can be within the budget it is given, and the stage before it should be recall-oriented — a candidate that never enters the funnel can never be ranked. Candidate generation optimises recall@k; ranking optimises precision at the top; re-ranking optimises the list.