Advanced
Open
Pro
Assembling a Turn's Context Budget
Your document assistant has a 200,000-token window. In early testing you notice two problems: (a) cost per query is higher than expected because the prompt-prefix cache almost never hits, and (b) on synthesis queries the model sometimes ignores clearly relevant passages that were retrieved and included in the context.
- Diagnose both problems in terms of what's in the prompt and in what order.
- Design the token budget for one turn: what segments make up the 200k window and roughly how large is each.
- State the ordering you'd use and explain why it fixes both problems at once rather than trading one for the other.
Share this question