Bootstrap Training Labels With an LLM Without Fooling Yourself

You have 200,000 unlabelled support tickets, 28 target classes, a budget for 1,500 human labels, and an LLM that can classify a ticket from a prompt. A colleague suggests: "Label all 200,000 with the LLM, split 80/20, train a classifier, report the test score."

1. Identify what is wrong with that plan.
2. Design a safer plan: how the 1,500 human labels are used, how many are needed to estimate the labeler's accuracy to within plus or minus 3 points if you expect 85 percent, and how you would detect systematic errors.
3. State how you would train on gold plus silver labels and what you would expect from the experiment in the lesson (a simulation with assumed labeler accuracy).
4. List the artefacts you would version and the checks you would repeat when the LLM, prompt or taxonomy changes.
5. Name the conditions under which the approach should not be used.