Prompt Caching and Context Cost Optimization

Ask an engineer who has only used a chat playground where an LLM app's money goes, and most guess "the long answers." Ask an engineer who has shipped one at volume, and the answer is almost always input tokens — the system prompt, the tool schemas, the retrieved chunks, the conversation history resent on every turn. Output is the part a user reads; input is the part your application silently re-pays for, over and over, on every single call. Interviewers probe this distinction because it separates candidates who have watched a bill from candidates who have only watched a demo: the fix for a runaway LLM cost line is almost never "make the model say less," it's "stop re-sending and re-computing the same context."

That reframes context engineering — how you assemble, order, and trim the tokens you send — as a cost-engineering discipline, not just a quality one. The same prompt layout decision that determines whether a model can find the right fact (covered in tokenization-and-context-windows and context-engineering-for-agents) also determines whether a provider can reuse computation it already did for you a moment ago. Move one sentence in a system prompt and you can silently multiply a production bill by 5-10x with no change in output quality at all — not because the tokens changed in count, but because they stopped being byte-identical to the previous call's tokens, and byte-identical is the only thing a cache checks for.

This subject goes deep on the mechanism specifically: how prefix caching and the underlying KV-cache reuse actually work, how to design prompts that hit the cache instead of silently missing it, and how caching composes with context trimming and model routing to produce order-of-magnitude cost reductions. cost-and-latency-engineering-for-llm-apps covers the broader cost/latency toolkit (semantic caching, batching, p95 latency budgets, cost-per-conversation accounting) — this subject is the focused deep dive on cache mechanics and how upstream context-engineering choices determine whether that cache actually gets hit.