The Cache Stopped Helping and Nobody Changed the Prompt Text
Your team's customer-support assistant uses a system prompt with prompt
caching enabled: a ~2,000-token block of role instructions, policies,
and tool schemas, followed by the per-request retrieved context and the
user's message. Cost per request has been stable for months at roughly
$0.004/request (mostly cache hits on the 2,000-token block). Last week,
someone added a small feature: the system prompt now opens with
"Today's date is {current_date}. You are a support assistant..." so
the model can reason about date-relative questions ("is my order still
within the return window"). No other text changed. This week, average
cost per request has risen to roughly $0.011, and nobody connected it
to the date change because "it's the same prompt, just one more fact
in it."
- Explain exactly why this one addition destroyed the caching benefit, in terms of how prefix matching actually works — not just "the prompt changed."
- Propose a fix that keeps the date-awareness feature but restores the cache-hit rate. Be specific about where the date field should live.
- Roughly estimate the cost impact using the numbers given: what fraction of the original savings did this one change erase, and why does even "one more fact" have an outsized effect on prefix caching specifically, more than it would on a token-budget alone?
1. Why the addition breaks the cache
Prefix caching matches a new prompt against a cached entry token by
token, starting from position zero, and the match ends at the first
point of divergence — everything after that point is treated as new
and recomputed at full price, regardless of whether the text after it
is identical to before. {current_date} sits at the very start of the
prompt and changes every single day (and in some implementations, the
field itself might be recomputed per-request even within a day if it's
templated freshly each time). Because it's the first thing in the
prompt, the divergence point is as early as it could possibly be — the
entire 2,000-token block that used to be a clean cache hit is now
100% recomputed on every request, even though 1,999 of those 2,000
tokens are byte-identical to yesterday's cached version. This is the
core mechanical point: cache matching is positional/prefix-based, not
"how much of the content is the same" — one early token invalidates
everything downstream of it regardless of how stable the rest is.
2. The fix
Move the variable date field to the end of the stable block (or into the variable suffix that follows the cached prefix entirely), so the 2,000 tokens of genuinely stable role/policy/tool-schema content stay as an unbroken, byte-identical prefix and only the small date field — plus everything that was already variable (retrieved context, user message) — falls outside the cached region:
[ CACHED PREFIX — unchanged ]
role instructions, policies, tool schemas (2,000 tokens, stable)
[ VARIABLE SUFFIX — was already uncached, now also holds the date ]
"Today's date is {current_date}."
retrieved context
user message
This restores the original cache-hit rate on the 2,000-token block while still giving the model the date fact on every request — the date doesn't need to be first for the model to use it correctly, it just needs to be present somewhere in context, and putting it in the variable suffix costs only a handful of tokens at full price per request instead of forfeiting the entire prefix's discount.
3. Cost impact estimate
Original cost ≈ $0.004/request (dominated by cache-read pricing on the 2,000-token block). New cost ≈ $0.011/request, roughly 2.75x the original — consistent with the entire 2,000-token block reverting from a steep cache-read discount (illustratively ~90% off) to full input price, since a discount that used to apply to essentially the whole request now applies to none of it. The reason "one more fact" has an outsized effect specifically on caching, more than it would on a token-budget calculation alone, is that a token-budget view treats every added token as a small, linear addition (2,001 tokens instead of 2,000 is a rounding error), while a cache-hit view treats where that token sits as the whole story — a token added at position 0 can flip 2,000 previously-discounted tokens to full price, while the identical token added at the very end of the same block would have cost almost nothing extra. Position, not token count, is the variable that determines caching's payoff.
Share this question