Setting a CI Regression Threshold That Doesn't Cry Wolf

Your team just wired up a CI eval gate: every PR touching a prompt runs the golden set and blocks the merge if the aggregate score drops at all versus the currently-deployed version. Within two weeks, engineers are complaining the gate fails on totally unrelated, cosmetic prompt changes (fixing a typo, rewording a comment) and several people have started re-running the CI job repeatedly until it happens to pass, rather than investigating failures.

1. Diagnose what's wrong with a zero-tolerance threshold here, mechanistically.
2. Describe how you'd determine a better threshold, including a concrete experiment you'd run first.
3. What's the risk of overcorrecting — setting the threshold too loose — and how would you know if you'd gone too far in that direction?