Intermediate
Open
Pro
A Prompt Change Regressed Production — Design the Process Gap
Post-incident review: an engineer edited the system prompt for your customer-facing chatbot directly in the deployed config to fix a complaint that answers were "too long," changing one sentence from "provide thorough, complete answers" to "be concise." The change shipped immediately (no PR, no review). Two days later, support tickets show a spike in "the bot refused to explain things" complaints — the model started truncating answers that genuinely needed detail (step-by-step account recovery instructions, for example), and nobody connected it to the prompt change until someone happened to check the config's edit history.
- List the concrete process gaps that let this happen, mapping each to what should have existed instead.
- Design the golden-set regression check that would have caught this before it shipped, including what the golden set needs to contain for a change like this specifically.
- Two days is a long time to notice a live regression. What observability would shorten that to hours, without requiring anyone to have predicted "conciseness" as the failure mode in advance?
Share this question