Cost and Latency Engineering for LLM Apps

"How would you make this cheaper?" and "how would you make this faster?" are two of the most concretely-gradeable questions in an AI Engineer interview, because unlike "design a RAG system" there is no room to hand-wave. The interviewer gives you token counts, prices and volumes, and either you can turn the crank — multiply, divide, compare levers — or you can't. This subject is that crank: the token-accounting model, the handful of levers that actually move cost and latency in production LLM systems, and the arithmetic that shows you did the multiplication rather than gestured at "caching helps".

The reason this topic rewards preparation more than most is that the levers are few, well-known and reusable across every LLM product: prompt-prefix caching, semantic caching, model routing, streaming, batching, and trimming what you send. What changes between systems is only the numbers you plug in. llm-application-system-design and tokenization-and-context-windows each cover pieces of this — token budgeting, the caching table, the routing paragraph — in the context of a broader design; the two case studies, case-study-customer-support-assistant and case-study-coding-agent, each work a full cost model for one specific system. This subject pulls the numbers apart from any one product and gives you the general model, so you can apply it cold to whatever system the interviewer puts in front of you.

A last framing point worth stating out loud in an interview: cost and latency are not the same optimization problem, even though they are often discussed in the same breath. Cost is driven by tokens times volume; latency is driven by the sequential path to a finished response. Some levers help both (routing to a smaller, faster model), some help one and are neutral to the other (streaming helps perceived latency and does nothing to cost), and a few actively trade one for the other (a bigger candidate set for re-ranking can improve quality at the cost of both). Knowing which bucket a lever belongs to, before you reach for it, is most of what separates a strong answer from a list of buzzwords.