Paths Subjects Questions Quizzes Pricing Search
Intermediate Open Pro

Should This Low-Traffic Endpoint Even Use Prompt Caching?

Your company runs two internal tools on the same LLM provider, both using a 3,000-token system prompt with a large set of tool schemas:

  • Tool A: an internal Slack bot used by ~5 employees, roughly 20 requests/day total, with long idle gaps between requests (often hours).
  • Tool B: a customer-facing chat widget handling ~50,000 requests/day, arriving continuously with no meaningful idle gaps.

A teammate proposes enabling explicit prompt-cache breakpoints on both, reasoning "caching only helps, so we should always turn it on."

  1. Using the general pricing shape (cache writes cost more than a standard input token; cache reads cost much less; cache entries expire after a short idle TTL), explain why this reasoning is wrong for at least one of the two tools.
  2. Work through the arithmetic: assuming input is priced at 3/1M tokens, a cache write at 1.25x that (3.75/1M), and a cache read at 0.1x that ($0.30/1M), compare the cost of the 3,000-token prefix under "always fresh, no caching" versus "caching enabled" for Tool A, given its request pattern.
  3. What would need to be true about Tool A's traffic pattern for caching to become worth enabling for it too?

Share this question

← Back to Prompt Caching and Context Cost Optimization practice

We use cookies for product analytics to improve OmniAtlas. See our Privacy Policy.