Triaging 40 Prompts Before Touching Any of Them
Your provider announces the model behind your production application will stop working in 60 days. The application has 40 distinct prompts built up over two years by several teams, with wildly different traffic and criticality, and almost none of them have a golden set. A colleague wants to start by opening the highest-traffic prompt file and rewriting it for the new model today.
- Explain what's wrong with starting there, and what "safe migration" should mean before any prompt is edited.
- Design the inventory/triage step: what axes you'd score all 40 prompts on, and how that scoring determines migration order.
- Give a plausible breakdown of the 40 prompts into a few triage buckets and say which bucket you'd migrate first, with reasoning.
1. Why "just start rewriting" is wrong
Rewriting before a baseline exists means there is nothing to detect a regression against — "looks fine" is not a migration gate. Safe migration means: no silent regression (changes are measured, not eyeballed), reversibility at every stage (rollback available until the old model is actually gone), and bounded blast radius (a bad migration of one prompt cannot take down an unrelated critical flow). None of that is possible if the first action is an edit.
2. Triage axes
Score every prompt on: traffic volume (determines statistical power available for eval and canary), blast radius/criticality (checkout or safety-relevant prompts need a higher confidence bar than an internal tool), existing eval coverage (most will have none — this sizes the Step 2 workload per prompt), prompt complexity/model-specific dependencies (heavy few-shot or brittle parsing built around old-model quirks is higher rewrite risk), and owning team (who signs off on the bar, who gets paged on a canary trip). Traffic × blast radius drives order; the other axes size the effort once a prompt is being worked.
3. Plausible buckets and ordering
Illustrative: 8 high-traffic/high-criticality, 15 high-traffic/ low-criticality, 17 long-tail (low traffic, mixed criticality). I'd migrate the 8 high-traffic/high-criticality prompts first — most value from getting it right, and the volume gives fast, statistically confident eval and canary reads, so the harder work happens while there's still runway to fix mistakes. (Migrating them last to "practice" on lower-stakes prompts first is also defensible if argued explicitly; migrating alphabetically or by whichever file is open is not, and is the actual trap this question is testing for.)
Share this question