Prompt-Prefix Caching in an Agent Loop
You are building a code-review agent that loops for an average of 25 steps per review (reading files, running linters, drafting comments). Each step sends 18,000 input tokens, of which 6,000 tokens are a stable prefix — system prompt, tool schemas and review guidelines that are byte-identical on every step — and produces 150 output tokens. Assume input is $2 per 1M tokens, output is $10 per 1M tokens, and a cached prefix is billed at 10% of the standard input price.
- Compute the cost of one review without any caching.
- Assume the prefix is a cache hit on steps 2 through 25 (24 of the 25 steps). Compute the cost of one review with prefix caching.
- At 50,000 reviews/day, what is the daily dollar saving from turning caching on?
- Why must the 6,000-token prefix be placed at the very start of the prompt, and be byte-identical every step, for any of this to work?
1. Baseline, no caching
- Input: 25 steps × 18,000 tokens = 450,000 tokens × $2/1M = $0.90.
- Output: 25 × 150 = 3,750 tokens × $10/1M = $0.0375.
- Total ≈ $0.9375 per review.
2. With prefix caching
The 6,000-token prefix repeats on all 25 steps; it is a cache hit on 24 of them (the first occurrence has nothing to hit). Those 24 steps would normally cost 24 × 6,000 = 144,000 tokens × $2/1M = $0.288 for just the prefix slice; at the 10%-of-standard cached price that slice costs 144,000 × $0.20/1M = $0.0288 instead. Saving = $0.288 − $0.0288 = $0.2592.
- New total ≈ $0.9375 − $0.2592 = $0.6783 per review (≈ $0.68) — roughly a 28% reduction from caching the prefix alone, with the rest of each step (12,000 non-prefix input tokens + output) unchanged.
3. Daily saving at 50,000 reviews/day
50,000 × 0.2592 ≈ **12,960/day** (≈ $13k/day, or roughly $390k/month) from this one lever, with zero change to review quality.
4. Why placement and exact-match matter
Provider-side prefix caching reuses the KV-cache computation for a prompt prefix that matches a previous call byte-for-byte from the start. If any variable content (today's date, a request ID, a differently-ordered tool list) sits before or inside the 6,000-token block, the match breaks at that point and everything after it is recomputed from scratch — you get none of the discount, not a partial one. Putting the stable material first and the variable material (the specific file being reviewed, this step's tool result) strictly after it is what makes 24 of 25 steps cache-eligible in the first place; it is a prompt-template design decision, not something the provider can fix for you.
Share this question