Model Serving & Deployment
Once a model clears offline evaluation, the interview question becomes: how does a prediction reach the user, how fast, at what cost, and what happens when the model is wrong or unavailable? This is where most ML system design candidates are weakest — they can describe a two-tower model in detail and then wave their hands at "deploy it behind an API". Interviewers at companies that run models at scale spend a large fraction of the session here, because serving is where the money is spent and where outages happen.
This subject covers the serving stage: choosing an inference mode, decomposing a latency budget, the shapes a model server can take, making inference cheaper (batching, caching, compression, hardware), rolling out safely (shadow, canary, blue-green, A/B, interleaving), keeping online features consistent with training, and degrading gracefully. Training and offline evaluation are covered in the Model Training & Experimentation at Scale subject; how you watch the deployed model for drift and regressions afterwards is covered in the ML Monitoring & Drift subject; the generic building blocks — load balancers, caches, queues, sharding, autoscaling groups — are covered in the System Design Interview track and are only referenced by name here.
The goal is that you can draw a serving architecture with a latency budget on it, defend every box, and explain how you would ship version 2 without anyone noticing unless it is better.
Inference Modes
There are four ways to get a prediction to a request. Pick deliberately; the choice sets latency, cost and freshness for the entire system.
| Mode | How it works | Latency at request | Freshness | Cost profile | Typical use |
|---|---|---|---|---|---|
| Batch (precompute & store) | Score all entities on a schedule (nightly/hourly), write results to a key-value store; request path is a lookup | ~1 ms (KV read) | Hours | Cheap; runs on spot/CPU offline; storage cost for the table | Churn scores, email recommendations, daily "top picks", lead scoring |
| Online / real-time | Compute the prediction on the request path with fresh features | 5–100 ms | Seconds (features) | Expensive; capacity for peak QPS, always on | Fraud checks, search ranking, ad CTR, chatbots |
| Streaming (event-driven) | Consume an event stream, score each event as it arrives, write results downstream | Sub-second after event | Seconds | Medium; sized for event rate not request rate | Anomaly detection on transactions, near-real-time personalisation updates, alerting |
| Hybrid | Precompute the expensive part (candidate lists, embeddings) offline; do a cheap online step (re-rank, filter, contextualise) | 10–100 ms | Mixed | Best of both if designed well | Recommendation feeds, notifications, ads |
Rules for choosing:
- Can you enumerate the inputs ahead of time? If the prediction depends only on the user (not the query, cart or context), batch is possible. If it depends on request-time context (search query, current session, live location), you need an online step.
- How stale is acceptable? Daily churn scores tolerate a day; fraud does not tolerate a minute.
- How many entities vs how many requests? Scoring 100 M users nightly when only 5 M visit tomorrow wastes 95 % of the compute; online scoring for 5 M visits may be cheaper. Conversely, scoring 5 M popular items for every request is absurd — precompute.
- Hybrid is the default answer for recommendation. Precompute user and item embeddings and a few hundred candidates per user; re-rank online with contextual features. It gets most of the freshness of online at a fraction of the cost.
Interviewers like hearing that batch is not "old-fashioned": it is the correct answer for a large class of problems and the cheapest system you will ever operate.