Intermediate
Open
Pro
Should This Low-Traffic Endpoint Even Use Prompt Caching?
Your company runs two internal tools on the same LLM provider, both using a 3,000-token system prompt with a large set of tool schemas:
- Tool A: an internal Slack bot used by ~5 employees, roughly 20 requests/day total, with long idle gaps between requests (often hours).
- Tool B: a customer-facing chat widget handling ~50,000 requests/day, arriving continuously with no meaningful idle gaps.
A teammate proposes enabling explicit prompt-cache breakpoints on both, reasoning "caching only helps, so we should always turn it on."
- Using the general pricing shape (cache writes cost more than a standard input token; cache reads cost much less; cache entries expire after a short idle TTL), explain why this reasoning is wrong for at least one of the two tools.
- Work through the arithmetic: assuming input is priced at 3/1M tokens, a cache write at 1.25x that (3.75/1M), and a cache read at 0.1x that ($0.30/1M), compare the cost of the 3,000-token prefix under "always fresh, no caching" versus "caching enabled" for Tool A, given its request pattern.
- What would need to be true about Tool A's traffic pattern for caching to become worth enabling for it too?
Share this question