Advanced
Open
Pro
Diagnosing Which Failure Mode Produced a Wrong Answer
A customer-facing agent has this sequence in one conversation:
- Turn 2: retrieval pulls a shipping policy doc that says "standard shipping: 5-7 business days."
- Turn 2 (same turn): the model, summarizing the policy back to the customer, says "5-7 days" (drops "business").
- Turn 9: the original policy document is no longer in the active context window (it aged out of the retrieval window for this follow-up question), but the earlier turn 2 assistant message ("5-7 days") is still in conversation history.
- Turn 9: the customer asks "so it'll definitely arrive by Friday?" (today is Monday) and the model confirms "yes, well within your 5-7 day window," incorrectly treating calendar days as if the original commitment held, compounding the earlier drop of "business."
- Name the failure mode(s) at play here, and be precise about where each one actually happened in the turn sequence — this scenario has more than one candidate answer and the numbered turns matter.
- Propose one concrete change to context assembly (not to the retrieval index or the underlying policy document) that would have prevented turn 9's answer from compounding the turn 2 error.
- Would simply keeping the original policy document in context for the entire conversation (never letting it age out) fully solve this? What does it fix and what does it not fix?
Share this question