A Support Bot's Context Budget Blows Up on a Single Customer
Your support bot has a 12,000-token practical per-turn budget, allocated roughly as: 1,200 system prompt + tool schemas (fixed), 4,000 retrieved policy content (capped, variable), 5,000 conversation history (capped, variable), 1,000 reserved for output, 800 margin. It has worked fine in testing.
In production, a customer with 300+ historical orders asks "what's the
status of all my recent orders?" The list_orders tool returns all 300
orders as raw JSON, which alone is around 18,000 tokens — blowing past
the entire budget before the model even sees the retrieved policy
content or gets to generate an answer. The request fails with a
context-length error.
- Diagnose which region of the context budget was actually unprotected, and explain mechanistically why this wasn't caught by the budget table above.
- Propose a fix for the
list_orderstool integration specifically — not a bigger budget. - Separately, is raising the practical budget from 12,000 to, say, 40,000 tokens a reasonable partial mitigation here? Justify with the mechanisms from context engineering, not just "more room is safer."
1. Which region was unprotected
The budget table names five regions but only imposes explicit caps on
"retrieved policy content" and "conversation history" — it never
assigns a cap to tool results, and the worked budget silently assumed
tool results would be small. A tool call is a variable region exactly
like retrieval or history, but because it wasn't named as one with its
own cap, nothing in the request pipeline filtered, truncated, or
summarized the list_orders output before it entered the context — the
tool's raw response size became the de facto region size, unbounded.
This is a specific instance of a general point: every source of
variable-length content needs an explicit cap and an explicit
overflow policy, and a source that wasn't in the original budget
exercise doesn't get a free pass just because it's a "tool result"
rather than "retrieval."
2. Fix for the tool integration
Treat list_orders the same way retrieval is treated: never insert the
raw result, insert a filtered/summarized version bounded by an explicit
cap. Concretely: (a) the tool itself should support pagination or a
limit/recent parameter so the model can request "last 5 orders" by
default rather than everything; (b) if a bulk result is genuinely
needed, the tool layer should summarize or aggregate before returning
to the model (e.g., "312 orders total; 5 most recent: [...]; 3 with
open issues: [...]" rather than all 312 rows); (c) define a hard token
cap on any single tool result (e.g., 1,500 tokens) with truncation and
a note to the model ("showing 5 of 312 orders; ask for more detail on
a specific order") when the cap is hit, so the failure degrades
gracefully into a smaller, still-useful answer instead of erroring the
whole request.
3. Is a bigger budget a reasonable mitigation
Only a weak, incomplete one, and it should not be presented as the
fix. Raising the budget to 40,000 tokens might make this specific
customer's 18,000-token dump fit, but it doesn't fix the underlying
problem: the tool result is still unbounded, so a customer with 3,000
orders reproduces the same failure at the new ceiling. It also
reintroduces the lost-in-the-middle and distraction costs from
tokenization-and-context-windows and this subject's selection
section — 300 raw order rows dumped into context is exactly the kind
of high-volume, low-precision material that dilutes the model's
attention on the policy content and the user's actual question, even
if it technically fits. The budget increase treats a selection problem
(the tool returns too much, unfiltered) as if it were a capacity
problem, and capacity increases are always the more expensive, less
targeted fix compared to bounding the source at the point where the
unbounded data enters the pipeline.
Share this question