Designing the Evaluation Harness and Shadow Phase

You are asked to prove, before launch, that the voice agent is safe and good enough. A first harness ran 1,200 simulated conversations (scripted personas through TTS, noise, accent and 8 kHz codec augmentation, scored against a sandbox backend) and reported task success 80.9 percent with a 95 percent bootstrap interval of [78.7, 83.1]. A new prompt version scored 83.0 percent on the same conversations. A teammate wants to ship it and skip the shadow phase.

1. Design the harness: personas, slices, metrics, how end state is checked, and what the zero-tolerance safety slice looks like.
2. Is 83.0 percent better than 80.9? What do you need to claim a 3-point improvement at 80 percent power and a 5 percent two-sided level, starting from a baseline near 0.80?
3. Per slice the success rates were 88.3, 86.4, 74.6, 74.5 and 69.4 percent (clean, light noise, heavy noise, accent group B, speakerphone), with 141 conversations in the accent slice. What do you conclude and what do you do?
4. What is the purpose of the shadow phase, what does 81.0 percent raw agreement with human outcomes tell you, and what can it not tell you?