Case Study: Design a Long-Context Document Assistant

"Design an assistant that can answer questions over our contracts / policies / technical manuals — about 10 million tokens of documents" is a different interview probe than "design a coding agent," even though both get called "AI system design." There is no filesystem to edit, no shell to run, no plan-act-verify loop — this scenario is deliberately not agent-shaped. It is a pure context engineering problem: how do you decide what a large language model actually sees on a given turn, when the source of truth is a hundred times bigger than any context window, the answer must be traceable to a specific passage, and the documents themselves cannot be trusted.

That is exactly why interviewers like it. It forces a candidate past "just use RAG" or "just use a long-context model" — both one-line answers that a strong candidate should recognize as incomplete — into the real design space: chunking that preserves citability, a retrieval strategy chosen by query shape, a token budget allocated on purpose, a compaction strategy that does not quietly destroy the evidence a citation depends on, and a threat model where the untrusted input is not the user but the corpus itself. chunking-and-embedding-strategies and guardrails-and-prompt-injection-defense are the deep dives behind two of the steps below; this case study is the model answer that assembles the pieces into one coherent system, the way an interview expects.

The single sentence to keep coming back to across every step: narrow the corpus down to what's relevant, then let the model read that relevant slice as coherent evidence it can be held accountable to. Almost every design decision below — chunking, retrieval, budgeting, compaction, grounding — is in service of that one sentence.