Case Study: Migrate a Production Prompt Suite Across a Model Deprecation

"Your model provider announces they're deprecating the model your production app runs on, effective in 60 days" is one of the most realistic AI Engineer interview scenarios there is — nearly every team running LLMs in production has lived through some version of it. It's a strong probe precisely because the tempting failure mode is visible from the first sentence: a candidate who starts swapping the model string and tweaking prompts by hand until outputs "look right" is demonstrating exactly the kind of production immaturity the question is designed to surface. The candidates who do well instead treat this as what it is — a migration of a system with unknown regression risk, on a deadline, with no rollback to the old model available once it's shut off — and reach for the machinery that makes a migration like that safe: evaluation, versioning, staged rollout, and monitoring that outlives the cutover date.

This subject is a model answer. It is the applied, ops-flavored payoff of two sibling subjects: prompt-evaluation-and-versioning covers the discipline of golden sets, regression suites and versioned prompts in the abstract, and prompt-caching-and-context-cost-optimization covers prefix layout and cache economics; this case study is what happens when both disciplines get exercised at once, under a real deadline, on a real production surface. Numbers throughout are illustrative assumptions, stated as such.