Sizing a Multi-Turn Support Prompt to a Token Budget
You are designing the prompt for a customer-support assistant. On a typical turn, the components are:
- System prompt + tool schemas: 850 tokens
- Few-shot examples (4 examples, ~180 tokens each)
- Retrieved help-center chunks: 7 chunks, ~480 tokens each
- Conversation history: the last 10 turns, averaging 220 tokens/turn
- Chat-template and special-token overhead: 150 tokens
- Current user message: 90 tokens
- Output reserve (
max_tokens): 400 tokens
Your team's practical per-request budget (chosen for cost and latency, not the model's advertised window) is 9,000 tokens total (input + output reserve).
- Compute the total token count for this prompt as specified.
- Is it within budget? If not, by how much is it over?
- Propose a concrete set of changes that bring it within budget, and compute the resulting total to show it works. Prefer changes that preserve answer quality over ones that blindly cut everything equally.
1. Compute the total
- System + tools: 850
- Few-shot: 4 × 180 = 720
- Retrieved chunks: 7 × 480 = 3,360
- History: 10 × 220 = 2,200
- Chat-template overhead: 150
- User message: 90
- Input subtotal: 850 + 720 + 3,360 + 2,200 + 150 + 90 = 7,370
- Output reserve: 400
- Total: 7,770 tokens
2. Within budget?
7,770 ≤ 9,000, so this prompt is technically within the stated 9,000-token budget — with about 1,230 tokens of headroom. A candidate who stops here without checking the arithmetic against the budget (or who assumes it must be over because the numbers look large) is missing the point of the exercise: the first job is to actually do the sum before proposing cuts.
3. Even though it fits, discuss what you'd do and why
Since the question invites a discussion of levers (and a real system would want headroom for edge cases — a longer user message, more chunks on an ambiguous query, more turns in a longer conversation — not just the average case), the recommended trims, in order of preference:
- History first: summarizing turns 1-5 into a ~150-token "conversation state" block instead of carrying all 10 turns verbatim cuts roughly 5 × 220 − 150 = 950 tokens, with the least quality impact — a summary of resolved earlier context is usually as useful as the verbatim text for a support conversation.
- Retrieved chunks second: dropping from 7 to 5 chunks (after confirming via retrieval eval that recall@5 is close to recall@7) saves 2 × 480 = 960 tokens and also reduces lost-in-the-middle risk from having too many chunks competing for attention.
- Few-shot last, and only if quality holds without it: dropping from 4 examples to 2 saves 2 × 180 = 360 tokens; this is the lever most likely to hurt quality, so it should be A/B tested rather than cut by default.
Applying just the first two (history summarization + fewer chunks): 7,770 − 950 − 960 = 5,860 tokens, which leaves ~3,100 tokens of headroom against the 9,000 budget — room to handle longer conversations or more ambiguous queries needing more chunks, without the average-case prompt already living at the edge of the budget. The key point to state out loud: budgeting to the average case with no headroom is itself a design mistake, because real traffic has a distribution, not a single fixed shape.
Share this question