LLM Application System Design
"Design a customer-support assistant on top of our help centre" is now one of the most common ML system design prompts, and it is a different interview from "design a CTR model". There is usually no training loop at the centre — the model is a hosted or open-weight LLM you call — so the interviewer is probing whether you can reason about retrieval, context, tokens, latency, cost, evaluation and safety as engineering quantities rather than as vibes. Candidates who treat the LLM as a magic box that "just answers" fail; candidates who can say "p95 latency is dominated by output tokens, so we stream and cap the answer at 300 tokens, and here is the cost per 1,000 conversations" pass.
This subject gives you the vocabulary and the decision frameworks: where LLMs fit relative to classic ML, when prompting alone is enough versus RAG versus fine-tuning, how a RAG pipeline is built end to end, how to budget tokens and money with real numbers, how to evaluate a system whose outputs are free text, how to guard it, when an agent is (not) justified, and how to observe it in production. It ends with a complete worked design and the traps interviewers set. The generic 7-step design framework, feature stores, model serving infrastructure and drift monitoring are covered in their own subjects in this track; here we apply them to LLM-specific concerns without re-teaching them.
Everything below is vendor-neutral. Where numbers appear (prices, throughput) they are stated as assumptions you should replace with your provider's actual figures — the arithmetic, not the constants, is what you are being tested on.
Where LLMs Fit vs Classic ML
An LLM is a general-purpose sequence model you condition with text. That makes it the right tool for some problems and an expensive, non-deterministic wrong tool for others.
| Problem shape | Prefer | Why |
|---|---|---|
| Structured tabular prediction (CTR, churn, fraud score) | Classic ML (GBDT, DNN) | Cheap, calibrated probabilities, millisecond latency, well-understood monitoring |
| Ranking millions of candidates per request | Classic retrieval + ranker | LLM per-candidate scoring is far too slow/expensive at scale; use LLMs offline to generate features or labels |
| Free-text understanding with little labelled data (summarise, extract, classify a long-tail of intents) | LLM (prompt or few-shot) | Zero/few-shot capability replaces months of labelling |
| Answering questions over a private corpus | LLM + RAG | Knowledge must come from your documents, not model weights |
| Generating natural language (drafts, replies, explanations) | LLM | Nothing else does this well |
| Multi-step tasks with tools (look up an order, then refund) | LLM agent (carefully) | Only if steps genuinely cannot be enumerated in advance |
Two patterns interviewers like to hear:
- LLM as offline labeller/feature generator. Use an LLM to label 100k examples or produce a "topic" feature, then train a small classic model to serve online. You get LLM quality at classic-ML latency and cost.
- Small model first, LLM as fallback. Route the 80% of easy cases to a cheap classifier or a small LLM; escalate the hard 20% (see routing below).
The framing to state out loud: "An LLM call costs roughly 3–5 orders of magnitude more than a GBDT inference and takes 100–1000× longer. I only put it on the request path where language understanding or generation is the product."