Paths Subjects Questions Quizzes Pricing Search
Intermediate Open Pro

Design a Routing Cascade and Justify the Threshold

Your team's document-classification endpoint currently sends every request to a large model at $12/1M input, $60/1M output. Average request: 800 input tokens, 40 output tokens. Volume: 2,000,000 requests/day. A small model is available at $2/1M input, $10/1M output, and on your golden set it matches the large model's accuracy on 85% of cases (determinable in advance by a lightweight confidence score the small model itself reports) but is meaningfully worse on the remaining 15%, which tend to be genuinely ambiguous documents.

  1. Design the cascade: what does the router actually do, and where does the confidence score come from?
  2. Compute the daily cost under three scenarios: (a) large model only, current state, (b) small model only, (c) the cascade — small model first, escalating the 15% low-confidence cases to the large model.
  3. The team is tempted to lower the confidence threshold further to push more requests to the small model and save more money. What would you check before agreeing to that, and why is "it saves more money" not sufficient justification on its own?

Share this question

← Back to Prompt Caching and Context Cost Optimization practice

We use cookies for product analytics to improve OmniAtlas. See our Privacy Policy.