Advanced
Open
Pro
Suspiciously Good Win-Rate — Check for Contamination
Part of the AI Engineer Interview path →
Part of the Reinforcement Learning & Long-term Optimization path →
This release's human-judged win-rate against the previous version came back at 78%, roughly double the 35-45% range typical of the last six cycles. The team is excited, but the eval-harness owner is suspicious.
- Before celebrating, what specific, mechanical check would you run first, and why is this result exactly the shape you'd expect from that specific failure mode (as opposed to, say, a genuinely excellent training run)?
- Walk through how the failure could have entered the pipeline, tracing it back through the stages described in this case's data pipeline.
- Assuming the check in (1) confirms the problem, what's the fix, and what would you add to the platform so this class of error is caught automatically next cycle instead of by a suspicious eval-harness owner?
Share this question