Redesigning a Tool That Confuses the Model
Your team shipped an internal assistant with one tool:
{
"name": "manage_ticket",
"description": "Handles ticket operations",
"parameters": {
"type": "object",
"properties": {
"action": { "type": "string" },
"data": { "type": "object" }
},
"required": ["action"]
}
}
In evaluation, the model regularly calls it with the wrong action
string (e.g. "update" when it meant "close"), and data comes back
shaped differently every time, which breaks the handler that parses
it.
- Explain why this tool's shape is causing exactly these two failure modes.
- Redesign it — show the new tool definition(s).
- What would you check in evaluation to confirm the redesign actually fixed tool-selection accuracy, not just made the schema prettier?
1. Why these failures happen
The model chooses tool behavior almost entirely by reading names,
descriptions and parameter schemas at inference time. manage_ticket
collapses several distinct intents (create, update, close, reassign)
into one free-text action string with no enum and no per-action
guidance, so the model has to guess the exact literal value with no
schema-level constraint — that's the "wrong action string" failure.
data is an untyped object, so nothing constrains what fields go
in it for a given action; the model invents a shape each time based
on the conversation, which is why the handler sees inconsistent
payloads. Both failures trace back to the same root cause: the tool
pushed a discrimination problem (which operation, with what fields)
into unconstrained string/object arguments instead of the schema.
2. Redesign
Split into one narrow tool per action, each with a tightly typed parameter list:
[
{
"name": "create_ticket",
"description": "Create a new support ticket. Use when the user reports a new issue that doesn't match an existing ticket.",
"parameters": {
"type": "object",
"properties": {
"summary": { "type": "string", "description": "One-line description of the issue" },
"priority": { "type": "string", "enum": ["low", "medium", "high"] }
},
"required": ["summary", "priority"]
}
},
{
"name": "close_ticket",
"description": "Close an existing ticket as resolved. Use only when the user confirms their issue is fixed. Do not use to cancel or reassign — use cancel_ticket or reassign_ticket instead.",
"parameters": {
"type": "object",
"properties": {
"ticket_id": { "type": "string" },
"resolution_note": { "type": "string" }
},
"required": ["ticket_id", "resolution_note"]
}
},
{
"name": "reassign_ticket",
"description": "Reassign an open ticket to a different team. Use when the current owner cannot resolve it.",
"parameters": {
"type": "object",
"properties": {
"ticket_id": { "type": "string" },
"new_team": { "type": "string", "enum": ["billing", "engineering", "account-management"] }
},
"required": ["ticket_id", "new_team"]
}
}
]
Each tool now answers one question ("which tool" rather than "which tool and which mode"), each has an enum or typed field wherever the value set is bounded, and each description states both when to use it and explicitly rules out the neighboring tool it's easiest to confuse it with — that negative case is cheap to add and directly reduces cross-tool confusion once there are several similar tools.
3. What to check in evaluation
Re-run (or build, if it didn't exist) an offline set of prompts with
a labeled expected tool and expected arguments, and measure: exact
tool-match rate, argument-match rate (not just "a tool was called"),
and the rate of calls with schema-invalid or empty required fields.
Also check for the specific old failure mode directly — a slice of
"ambiguous action" prompts that previously produced the wrong
action string — to confirm those now resolve to the correct tool.
A redesign that "looks cleaner" but doesn't move exact tool-match
and argument-match rates on a held-out set hasn't actually fixed
anything; schema tidiness is not the same as measured tool-selection
accuracy.
Share this question