ML System Design Interview Framework

The ML system design interview is where senior ML roles are decided. Coding rounds test whether you can implement; this round tests whether you can be trusted with an ambiguous business problem — "design the feed", "detect fraud", "recommend videos" — and turn it into a working, measurable, maintainable system. Most candidates fail it not for lack of ML knowledge but for lack of structure: they hear "recommend videos", say "two-tower model", and spend 30 minutes on architecture while never saying what the label is, how the model is evaluated, or what happens when it degrades in production.

This subject gives you the skeleton every later subject in this track hangs on: a 7-step framework, a 45-minute time box, the estimation arithmetic interviewers expect you to do on the spot, the standard problem framings, and a checklist to memorise. Each later subject (data pipelines, training, serving, monitoring, ranking, LLM systems, and the four case studies) deep-dives one step of this framework — here we build the map, not the territory.

One boundary up front: this track assumes you know the generic distributed-systems toolkit — load balancers, caches, sharding, queues, CAP — from the System Design Interview track. We reference those in one line where they matter and stay ML-specific.

How the ML Interview Differs From the Classic One

A classic system design interview asks you to build a deterministic system: given the same request, the same output. Success is measured in latency, throughput, availability, and cost. An ML system design interview asks you to build a system whose core component is learned from data, so its correctness is statistical, its quality decays over time, and it can be wrong in ways nobody wrote code for.

What interviewers actually score

Rubrics vary by company but converge on five signals:

1. Problem framing — did you convert the business ask into a precise ML task with a defined label, prediction target, and success metric?
2. Data reasoning — do you know where labels come from, how noisy/delayed they are, and how you would construct a training set without leakage?
3. Modelling judgement — did you propose a baseline first, then justify complexity? Can you name trade-offs (latency vs accuracy, freshness vs cost)?
4. Evaluation rigour — offline metric aligned with the business objective, plus a plan for online experiments.
5. Production thinking — serving path, latency budget, monitoring, retraining, failure handling.

A candidate who nails 1, 2 and 4 with a logistic regression usually out-scores one who draws a transformer and skips the label discussion. Interviewers are asking themselves: "Would this person ship something that works and keeps working?"

The 7-Step Framework

Memorise the headings. In the interview, say them out loud — narrating the structure is itself a signal.

 ┌──────────────────────────────────────────────────────────────────┐
 │ 1. Clarify requirements & constraints                            │
 │ 2. Business objective  →  ML objective                           │
 │ 3. Data: sources, labels, sampling                               │
 │ 4. Features & feature pipeline                                   │
 │ 5. Model choice & baselines                                      │
 │ 6. Offline evaluation & metric choice                            │
 │ 7. Serving, online eval (A/B), monitoring & retraining           │
 └──────────────────────────────────────────────────────────────────┘
      Steps 1–2: what are we building?      Steps 3–6: the model
      Step 7: keeping it alive in production

Step 1 — Clarify requirements & constraints (5 min)

Never start designing from the prompt as given. Ask, and write down the answers:

• Users & product surface: who sees the output, on what screen, how often?
• Scale: DAU/MAU, requests per second, catalogue size (items, ads, documents).
• Latency budget: is the prediction on the request path (< 100 ms end-to-end) or batch (nightly)? This decides almost everything downstream.
• Online vs offline: real-time scoring, near-real-time (minutes), or precomputed?
• Cost & hardware: is a GPU fleet acceptable, or is this CPU-only at the edge?
• Privacy / compliance: is user data restricted (GDPR, HIPAA, minors)? Can we log what we need?
• Existing system: is there a heuristic in place today? What is its baseline number?

Two or three targeted questions are enough. State assumptions explicitly for the rest ("I'll assume 50 M DAU and a 200 ms p99 budget for the whole page, so ~50 ms for our model").

Step 2 — Translate the business objective into an ML objective (3 min)

This is the step candidates skip most and interviewers weigh most. Business goals are not predictable quantities; you have to pick a proxy that is.

Say the proxy, name its gap from the true goal, and how you'd guard the gap (e.g. combine watch probability with a completion term). Also decide the prediction unit (per user? per user–item pair? per session?) and the decision the prediction drives (rank, threshold, allocate budget).

Step 3 — Data: sources, labels, sampling (7 min)

• Sources: event logs (impressions, clicks, purchases), user profile tables, item metadata, third-party signals.
• Labels: explicit (ratings, thumbs-up, "report") are clean but sparse; implicit (click, dwell, purchase) are abundant but noisy. Ask: how delayed is the label (a chargeback arrives 30–90 days later)? How noisy (accidental clicks)? Is there selection bias (you only have labels for items you showed)?
• Sampling: for a 0.1 % positive rate you rarely train on every negative — downsample negatives and re-calibrate. For implicit feedback (recommendations), you often have no explicit negatives, so you generate them: in-batch negatives, random negatives, or "impressed but not clicked".
• Splits: split by time, not randomly, when the deployment is forward in time.

The mechanics — point-in-time joins, leakage, feature stores — are covered in the ML Data Pipelines & Feature Stores subject.

Step 4 — Features & feature pipeline (5 min)

Group features by entity and freshness:

User features     : demographics, long-term aggregates (30-day CTR), embeddings
Item features     : category, age, popularity counters, content embeddings
Context features  : time of day, device, position/slot, query
Cross features    : user×item history (has this user seen this creator?),
                    similarity(user_emb, item_emb)
Real-time features: last 10 events in this session (streaming)

State which are computed in batch (nightly aggregates) versus streaming (session counters), and that the same code path must produce features at training and serving time to avoid training–serving skew. Mention the feature store as the mechanism, then move on.

Step 5 — Model choice & baselines (7 min)

Always climb the ladder; never start at the top.

Interviewers want the reasoning: "I'd start with a GBM on tabular features because we have 10⁷ labelled rows and a 20 ms budget; I'd move to a two-tower model for retrieval because we need to score 10⁶ items in under 10 ms, which only works with approximate nearest-neighbour search over embeddings." Mention what the baseline is, and what improvement would justify each step up.

Step 6 — Offline evaluation & metric choice (5 min)

Pick a metric that matches the decision:

Also state: temporal validation split, a comparison against the baseline, and slice-level metrics (new users, cold-start items, each country). ROC-AUC alone on a 0.1 % positive-rate problem is a red flag — say PR-AUC.

Step 7 — Serving, online evaluation, monitoring & retraining (8 min)

• Serving path: batch precompute (write scores to a KV store) vs online inference (model server behind the API); embedding lookup + ANN index for retrieval; feature fetch from an online store; model artefact versioning and rollback.
• Online evaluation: A/B test on the business metric (watch time, revenue, fraud loss), guardrail metrics (latency, complaints), interleaving for ranking. Offline gains do not always transfer — say so.
• Monitoring: prediction distribution, feature distribution drift, label-delay-aware quality tracking, calibration over time, latency and error rates.
• Retraining: cadence (daily/weekly/continuous), triggers (drift alarm, metric drop), champion–challenger promotion.

Depth on each lives in the Model Serving & Deployment and ML Monitoring & Drift subjects; here you need one confident sentence per bullet.

The 45-Minute Plan

 min  0– 5  Step 1  Clarify + assumptions (write them top-left on the board)
 min  5– 8  Step 2  Business → ML objective; label; decision
 min  8–15  Step 3  Data & labels; sampling; split strategy
 min 15–20  Step 4  Features + high-level pipeline diagram
 min 20–27  Step 5  Baseline → model ladder; justify the chosen rung
 min 27–32  Step 6  Offline metrics + validation
 min 32–40  Step 7  Serving diagram, A/B, monitoring, retraining
 min 40–45  Deep-dive on whatever the interviewer pulls on; trade-offs; wrap-up

Rules of the clock:

• If you are past minute 15 and have not said the word "label", stop and say it.
• Draw one end-to-end diagram (data → features → training → model store → serving → logs → back to data) by minute 20; annotate it rather than drawing new ones.
• Leave the last 5 minutes. Interviewers score depth-on-demand; you need slack to go deep on one component they pick.

Back-of-Envelope Estimation for ML

You will be asked to size things. Know the constants and the arithmetic.

Useful constants

• Seconds per day \approx 86{,}400 \approx 10^5 (round up).
• One float32 = 4 bytes; float16/bfloat16 = 2 bytes; int8 = 1 byte.
• Rough per-item serving cost of a GBM on CPU: ~0.1–1 ms per 1,000 trees; a small MLP scores a batch of hundreds in ~1 ms on CPU.
• A modern datacentre GPU does on the order of 10^{14}–10^{15} FLOP/s in low precision; treat any specific number as an assumption you state.

Worked example — video recommendation feed

Assumptions: 50 M DAU, each user opens the feed 10 times/day, each open requests 1 ranked page.

• Requests/day = 50 \times 10^6 \times 10 = 5 \times 10^8.
• Average QPS = 5 \times 10^8 / 10^5 \approx 5{,}000; peak ≈ 3× → ~15,000 QPS.
• Each request ranks 500 candidates → 7.5 M item-scorings per second at peak. That immediately says: retrieval must be ANN over embeddings, and the ranker must be cheap per item or batched on accelerators.

Feature bytes: 200 features × 4 bytes ≈ 800 B per (user, item) pair before compression. Fetching user features once (~1 KB) and item features for 500 candidates (500 × 400 B = 200 KB) per request → at 15 k QPS that is ~3 GB/s of feature reads. Conclusion: item features must be cached in-process or served from an in-memory online store colocated with the ranker; the System Design Interview track's caching material applies directly.

Embedding table: 10 M items × 64-dim × float32 = 10^7 \times 64 \times 4 = 2.56 \times 10^9 bytes ≈ 2.6 GB — fits in one machine's RAM, replicate it. 500 M users × 64 × 4 = 128 GB — shard by user id, or use hashed/compressed embeddings, or compute user embeddings on the fly from a small tower.

Training data volume: log 1 % of impressions → 5 \times 10^8 \times 500 \times 0.01 = 2.5 \times 10^9 rows/day. At ~1 KB/row raw that is 2.5 TB/day; 30 days ≈ 75 TB. Conclusion: columnar storage, feature logging at serve time, and negative downsampling are requirements, not nice-to-haves.

Inference cost sanity check: suppose the ranker is a small neural net at ~2 M FLOPs per (user, item). 7.5 \times 10^6 \times 2 \times 10^6 = 1.5 \times 10^{13} FLOP/s at peak. On paper that is a fraction of one GPU; in practice with 30–50 % utilisation and redundancy across regions you would provision a handful of GPUs, or run it on CPU with batching. The point of the estimate is the order of magnitude and the design consequence, not the exact number.

Say the arithmetic out loud, round aggressively, and end each estimate with "…so that means we need X". Estimation without a design consequence is wasted time.

Common ML Problem Framings

Most prompts collapse into one of five framings. Recognising which one you are in — fast — tells you the label, model family, and metric.

How to spot which one:

• Is the output a list from a large pool? → retrieval + ranking (two stages, not one).
• Is the output a decision on one item? → classification (and the operating threshold is part of the design).
• Is the output a number with units? → regression (ask about the loss: MSE punishes big misses, MAE is robust).
• Is the output free-form content? → generation (and evaluation is your hardest problem).

Many real prompts are compositions: search = retrieval → ranking → (optional) generation of snippets. Say the decomposition.

The Candidate Generation → Ranking → Re-ranking Pattern

Any time the answer is "pick k items from millions", the answer is a funnel:

   Catalogue                Candidates             Scored               Final page
   10^6–10^9 items   ───►   ~1,000 items   ───►   ~1,000 with     ───►   ~20 items
                     ANN /                  heavy   P(click),      rules,
                     CF /                   ranker  P(watch≥30s)   diversity,
                     rules                                          business
                                                                     constraints
   ─────────────────  cheap, high recall  ─────────  expensive, precise  ─────────

• Candidate generation: several cheap sources in parallel (embedding ANN, collaborative filtering, "trending", subscriptions), each optimised for recall.
• Ranking: one heavier model over the union of candidates, rich cross features, optimised for the ML objective from Step 2 (often multi-task: click, watch, like).
• Re-ranking: deterministic-ish policy layer — diversity, dedup, freshness quotas, safety filters, sponsored slots.

Preview only: mechanics of two-tower models, ANN indexes, multi-task ranking and position bias are covered in the Ranking & Recommendation Systems subject. In this interview framework, your job is to name the funnel and place your latency budget across the three stages.

Common Candidate Mistakes / Interview Traps

• Jumping to the model. Saying "transformer" before saying "label" is the single most common failure. Fix: Steps 1–3 before any architecture word.
• Ignoring labels. Treating labels as given. Ask how delayed, how noisy, how biased by what was shown. Fraud labels arrive months late; click labels have position bias.
• Ignoring latency. Proposing a cross-attention model over 10⁶ items for a 50 ms budget. Every model choice must survive the funnel and the budget you stated in Step 1.
• No baseline. Without a heuristic or LR baseline you cannot claim your model helps and you cannot detect regressions. State the baseline and its expected metric.
• No monitoring / retraining story. A model that is never retrained decays; a model that is never monitored decays silently. One sentence each is enough — zero sentences is a fail.
• Wrong metric. Accuracy at 0.1 % positive rate; ROC-AUC when precision at the top matters; NDCG when the downstream consumer needs calibrated probabilities.
• Random train/test split on temporal data. Leaks the future. Say "time-based split" unprompted.
• Estimation with no consequence. Computing 15 k QPS and then not letting it change the design.
• Feedback loops unmentioned. Recommenders train on their own outputs; fraud models change fraudster behaviour. Mention exploration / logging of randomised traffic.
• Talking only. Draw one diagram, write assumptions on the board, keep the interviewer oriented.

One-Page Checklist to Memorise

REQUIREMENTS   users · scale · latency budget · online/offline · cost · privacy · existing baseline
OBJECTIVE      business goal → predicted quantity → decision → success metric → known gap
DATA           sources · label type (explicit/implicit) · delay · noise · bias ·
               negatives · sampling · time-based split
FEATURES       user / item / context / cross / real-time · batch vs streaming ·
               same code at train & serve · feature store
MODEL          heuristic → LR → GBM → deep · why this rung · latency fits budget ·
               retrieval vs ranking stages
OFFLINE EVAL   metric matches decision · slices · calibration if probabilities are consumed
SERVING        batch vs online · model store & versioning · fallback · ANN for retrieval
ONLINE EVAL    A/B on business metric · guardrails · interleaving for ranking
MONITORING     feature drift · prediction drift · delayed-label quality · latency
RETRAINING     cadence · triggers · champion/challenger · rollback

Practise until you can reproduce this in 60 seconds from memory. It doubles as your whiteboard layout: write the left column top-to-bottom at minute 0, fill it in as you go.

Eight Example Prompts and How to Frame Each

For every prompt: say the framing, the label, the metric, and the funnel in your first two minutes. Then run the seven steps.

Key Takeaways

• ML system design is scored on framing, data, evaluation and production thinking — architecture is necessary but not sufficient.
• Run the 7 steps in order: requirements → ML objective → data & labels → features → baseline-first models → offline metrics → serving/A/B/monitoring/retraining. Say the headings out loud.
• Translate the business goal into a predictable label and a decision; name the gap between proxy and goal.
• Labels are the design. Ask how they are collected, delayed, noisy and biased before proposing any model.
• Climb the model ladder — heuristic → LR → GBM → deep — and justify each rung against data volume and latency budget.
• Estimation must end in a design consequence: QPS ⇒ funnel; embedding bytes ⇒ shard or replicate; training rows ⇒ logging and sampling strategy.
• Recognise the framing (classification, regression, ranking, retrieval, generation) and the candidate generation → ranking → re-ranking funnel.
• Never end without A/B testing, monitoring and a retraining plan — that is what separates a demo from a system.