Paths Subjects Questions Quizzes Pricing Search
Advanced Open Pro

Cost per 1,000 Conversations, Lever by Lever

A coding-assistant chat product averages 5 turns per conversation. Each turn sends 4,000 input tokens (1,200 of which are a stable, cache-eligible prefix: system prompt and tool schemas) and produces 220 output tokens. Assume input is $0.80 per 1M tokens, output is $2.40 per 1M tokens, and a cached prefix is billed at 20% of the standard input price.

  1. Compute the baseline cost per 1,000 conversations with no optimization.
  2. The prefix is a cache hit on turns 2–5 of every conversation (4 of 5 turns). Recompute cost per 1,000 conversations with prefix caching applied.
  3. Separately, suppose 55% of conversations are simple enough to be routed entirely to a small model priced at one-sixth of the large model's rates (apply this on top of the caching from part 2, i.e. the small-model conversations also benefit from caching on their own prefix at the same 20%-of-small-input-price ratio). Compute the new blended cost per 1,000 conversations.
  4. In one or two sentences, state which of the two levers did more work here, and why that ordering (cache first, then route) is also the right order to implement them in.

Share this question

← Back to Cost and Latency Engineering for LLM Apps practice

We use cookies for product analytics to improve OmniAtlas. See our Privacy Policy.