Model Training & Experimentation at Scale
In an ML system design interview, "how do you train the model?" is rarely about the loss function. The interviewer wants to hear a plan: what you build first, how you decide whether it works before you ship it, how you keep the experiment honest, and what it costs to retrain it every day at production scale. Candidates who jump straight to "I'd fine-tune a transformer" without a baseline, a temporal split and a retraining story lose points even if the model choice is right.
This subject covers the training and experimentation stage of the ML lifecycle: from baseline to tuned model, from a single laptop run to a distributed job over a billion events, and from an offline AUC number to a defensible claim about online impact. Data ingestion and feature computation are covered in the ML Data Pipelines & Feature Stores subject; how the trained artifact is served is covered in the Model Serving & Deployment subject; the definitions of the metrics themselves (AUC, log loss, precision/recall, NDCG) are covered in the Data Science track's Model Evaluation Metrics subject. Generic distributed-systems topics (queues, sharding, load balancing) belong to the System Design Interview track and are only referenced here.
By the end you should be able to sketch a complete training plan on a whiteboard, with numbers, in about ten minutes.
Baselines First
The first model you train should be the dumbest one that could plausibly work. There are three reasons, and you should be able to say all three:
- A baseline calibrates every later number. An AUC of 0.78 means nothing until you know that "predict the global click rate" gets 0.50 and logistic regression on ten features gets 0.74. The delta over the baseline is what you are paying for.
- A baseline debugs the pipeline. If popularity ranking cannot beat random on your offline metric, your labels, joins or splits are broken. Better to discover that on a model that trains in one minute.
- A baseline is the fallback. When the real model is down, times out or drifts, the serving layer degrades to the baseline. If you never built one, the fallback is "return nothing".
The usual ladder:
| Level | Baseline | Typical use |
|---|---|---|
| 0 | Constant / random | Sanity check of metric code |
| 1 | Heuristic rule ("most recent", "same category") | Product intuition, cold-start fallback |
| 2 | Popularity / global rate per segment | Recommendation, CTR, search |
| 3 | Logistic regression or a small GBDT on hand features | First "learned" model; often shipped |
| 4 | Full model (deep, sequence, pretrained) | Only if the delta over 3 justifies the cost |
Levels 2 and 3 are the ones interviewers expect you to name explicitly. Popularity is embarrassingly hard to beat for new users; logistic regression is cheap, interpretable, and its coefficients tell you which features carry signal before you invest in feature engineering for a bigger model.