Case Study: Design a Customer-Support Assistant
"Design an AI assistant for our customer support" is the most common AI Engineer design prompt, and it is deceptively open. A weak answer describes a chatbot over a vector database. A strong answer treats it as a product with a failure budget: it decides what the assistant is allowed to do, grounds every answer in retrievable facts, keeps tenants apart at the retrieval layer, gives the model narrow tools with a human confirmation step for anything irreversible, defines the hand-off to a person, evaluates the whole thing in layers, and puts a dollar figure and a latency figure on the design before the interviewer asks.
This subject is a model answer. The LLM Application System Design subject covers the underlying concepts — prompt vs RAG vs fine-tuning, RAG internals, token budgeting, evaluation, guardrails — and ends with this design in compressed form. Here it is expanded into the full 40-minute answer, with the follow-up questions interviewers actually ask. Where a number appears it is an assumption to state, not a fact to memorise; you are graded on the reasoning.
Step 1 — Clarify Requirements and Non-Goals
Spend the first five minutes here. The prompt is under-specified on purpose.
Questions to ask
- What can the assistant do: only answer questions, or also act (look up an order, change a plan, issue a refund)? This is the biggest fork — see Step 2.
- Who are the users: end customers (unauthenticated + authenticated), or internal support agents (an "agent-assist" copilot)? Assume end customers, both anonymous and logged in.
- Where does knowledge live: a help centre, past resolved tickets, internal runbooks, product database? Assume a help centre of ~5,000 articles updated daily, plus account data behind internal APIs.
- Multi-tenant? Assume a B2B SaaS with thousands of customer organisations, each with its own configuration and data — cross-tenant leakage is a hard failure.
- Channels: web widget, email, in-app? Assume web chat; note that email is asynchronous and changes the latency story.
- Languages, regions, compliance (PII, data residency)? Assume English first, PII present in conversations.
- What is "success"? Assume the business metric is deflection (conversations resolved without a human) without hurting CSAT.
Assumptions to state
| Quantity | Assumption |
|---|---|
| Conversations | 2 M/day, ≈ 4 turns each → ≈ 8 M LLM turns/day |
| Peak | ≈ 3× average → ~300 turns/s |
| Knowledge base | 5,000 articles, ~50 k chunks, updated daily + on publish |
| Latency target | first token < 1 s p95; full answer < 6 s |
| Grounding | every factual claim cites a help-centre article or an account lookup |
| Actions | read-only lookups freely; mutating actions (refund, plan change) require explicit user confirmation and policy checks |
| Hand-off | human agents available; escalation must carry the full context |
| Budget | LLM spend cap per month; cost per conversation must be tracked |
Non-goals (say them): not a general chatbot; does not answer outside the product; does not make promises about billing that the policy engine would not; does not replace the human team on day one.
The two facts that shape the design: the assistant can act, not just answer, and it serves many tenants from one system.