Validate LLM-Produced Labels Before Using Them

Your team labels 100,000 support messages with a prompted large language model and proposes to train a classifier on the labels and to evaluate it on a further 5,000 LLM-labeled messages to save money. Assume (as a placeholder) that the model is 88 percent accurate against careful human labels, and that a human expert label costs USD 1.20.

1. Say what is wrong with the evaluation plan and what you would do instead.
2. Compute the number of human-verified items needed to bound the LLM's accuracy to plus or minus 2 points at 95 percent confidence, and its cost. What does a 300-item check give?
3. Describe a verification design that also covers the rare classes and the error structure.
4. Say how you would use the LLM labels safely in training, and what you would monitor over time.