An Evaluation Plan Leadership Will Trust

Your VP asks: "How will we know the assistant is good enough to turn on for customers, and how will we know it is still good next quarter?" Design the evaluation plan for the support assistant. Cover the offline gold set and its composition, the metrics per layer with example gates, how you use LLM-as-judge without being fooled by it, the online experiment and its guardrail metrics, and how the two loops feed each other over time.