Building a Golden Set From Scratch for a Refund-Request Classifier
You're taking over a prompt that classifies refund requests into
approve, deny, or escalate_to_human. There is currently no golden
set at all — the previous owner "tested" changes by pasting a handful
of examples into a playground and eyeballing the output before shipping.
You have access to six months of production logs (roughly 40,000
requests) with no labels, plus a support-ticket archive showing which
requests were later overturned on appeal.
- Describe concretely how you would build the first version of the golden set from this raw material — what you'd sample, how much, and what you'd deliberately over-represent.
- What labeling process would you put in place to avoid the common pitfalls of inconsistent or self-referential labels?
- Six months from now, how do you know if the golden set has gone stale, and what would you do about it?
1. Sampling strategy
Don't sample uniformly at random from the 40,000 — that mostly captures easy, unambiguous cases and under-represents the tail where the prompt actually needs to be correct. Instead: stratify by whatever categories or amounts the business cares about (refund size bucket, product line, customer tenure) to get baseline coverage of the normal distribution, then deliberately over-sample two additional slices: requests that were later overturned on appeal (pulled directly from the support-ticket archive — these are confirmed cases where the original decision was wrong, exactly the failure mode a golden set needs to catch), and requests near an obvious decision boundary (large refund amount close to a policy threshold, ambiguous free-text justification). A first version in the 100–150 example range is enough to hand-label carefully and still catch a meaningful regression, growing opportunistically after that rather than trying to label thousands up front.
2. Labeling process
Write the rubric before labeling anything: define explicitly what
makes a request approve vs deny vs escalate_to_human in policy
terms (not "what feels right"), including how to handle the ambiguous
cases the sampling step over-represented. Label against the actual
policy and the overturned-appeal ground truth, not against outputs the
current prompt produces — generating candidate labels from the
existing prompt and lightly correcting them would anchor the golden
set to the current prompt's blind spots instead of to what's actually
correct. Where two labelers disagree, that's a sign the rubric itself
is ambiguous on that case — resolve it by refining the rubric, not by
picking a tie-breaker and moving on, since the same ambiguity will
recur in production traffic.
3. Detecting and fixing staleness
Staleness shows up as a growing gap between golden-set performance and real-world outcomes: a rising rate of appeals/overturns on requests the current prompt classified in a way the golden set would have scored as correct, or a refund policy change that the golden set's expected labels no longer reflect. Concretely: review the golden set on the same cadence the refund policy itself is reviewed (a policy change should trigger a golden-set audit, not just a prompt audit), and treat every newly overturned appeal as a candidate golden-set addition — if the prompt's decision was wrong in a way the golden set didn't already cover, that's a gap in the golden set, not just a gap in the prompt. A golden set that hasn't grown or changed in six months on a policy that has changed is very likely stale even if nobody has measured it directly yet.
Share this question