Designing a CI Regression Gate for Prompt and Model Changes

Your team currently ships prompt changes by having an engineer try a handful of examples manually, decide "it feels better," and merge. You've been asked to replace this with a CI gate that runs before any prompt, model, or retrieval-config change can merge.

1. Design the gate: what runs, what it's checked against, and what specifically blocks a merge.
2. Explain concretely why "it felt better in a few manual tests" is not sufficient evidence, in terms of what a small number of manual tries can and cannot tell you.
3. A teammate proposes that provider model upgrades (e.g. the provider silently updates what a version alias points to) don't need to go through this gate because "we didn't change anything." Respond to that claim.
4. How does this offline gate relate to the canary rollout you'd still do after merge — why isn't the gate alone sufficient?