Design a Labeling Program for 100,000 Messages on a Fixed Budget

1. Order of work. Build things in the order in which mistakes get expensive. (a) Write a one-page guideline with definitions, positive and negative examples per class and an edge-case list, and give annotators an explicit "unsure / none of these" option. (b) Run a calibration batch of 50 to 100 items with every annotator, compute Cohen or Fleiss kappa and per-class agreement, and fix the guideline where they disagree; repeat until the disagreements left are genuinely ambiguous items rather than unclear rules. (c) Build the gold set: a random sample of the live distribution, labeled by three in-house annotators and adjudicated by a senior reviewer. It doubles as the frozen test set and as the source of honeypots. (d) Label training data with the cheap channel (crowd) in queues that contain 5 percent honeypots, and re-label 10 percent by two more annotators for a running agreement estimate. (e) Version the guideline and store the source, guideline version and timestamp with every label.

Detecting bad annotators. Honeypot accuracy per annotator, tracked weekly, plus label-rate drift against the pool average. Power matters: flagging anyone below 0.70 honeypot accuracy catches a 0.60-accuracy annotator with probability 0.618 after 10 honeypots, 0.871 after 40 and 0.994 after 160, while a good 0.85 annotator is falsely flagged with probability 0.050, 0.004 and about 0. Honeypots catch persistent problems, not a bad afternoon.

2. Budget arithmetic. A gold item costs three in-house labels (3 \times 1.20 = 3.60) plus an adjudication with probability 0.47 (the simulated any-disagreement rate for three annotators at roughly 0.80 accuracy; two annotators disagree on 0.356), so 0.47 \times 2.00 = 0.94 and the cost per gold item is about USD 4.54.

• 300 gold items: 300 \times 3.60 = 1{,}080 plus 141 \times 2.00 = 282, so USD 1,362. Remaining 3{,}000 - 1{,}362 = 1{,}638. A crowd training item costs one label (0.30), plus 10 percent re-labeled twice (0.2 \times 0.30 = 0.06), plus 5 percent honeypots (0.05 \times 0.30 = 0.015), which is 0.375 per item. So 1{,}638 / 0.375 \approx 4,368 training items.
• 500 gold items: 500 \times 3.60 = 1{,}800 plus 235 \times 2.00 = 470, so USD 2,270. Remaining 730, so 730 / 0.375 \approx 1,947 training items.

The sticker price of 0.30 per label hides all of this: the effective cost per usable training label is 0.375 here, and the gold set costs about 12 times more per item.

3. Choosing the gold size. The macro-F1 of one fixed classifier had a standard deviation of 0.047, 0.030, 0.020 and 0.008 over random test sets of 200, 500, 1,000 and 5,000 items, which scales roughly as 1/\sqrt{n}. A 300-item set (about 0.039 by that scaling) is too noisy to rank two models that differ by two or three points, and it holds only about 6 items of the 2 percent class, so per-class results are anecdotes. The 500-item set (std 0.030, about 10 rare-class items) costs 4{,}368 - 1{,}947 = 2{,}421 crowd training items. That is a real price, but the crowd labels are noisy and the test set decides every later decision, so I choose 500 gold items, add a stratified top-up of the rare classes if the budget allows, and fill the rest with active-learning or weak-supervision labels (see the other questions).

4. What to measure. Cohen or Fleiss kappa and raw agreement per class and per batch; honeypot accuracy per annotator per week; the fraction of label events that are overhead (here a third of events); cost per usable label; the learning curve (test macro-F1 against labels) to see when more labels stop paying; and the confusion matrix of out-of-fold predictions against labels to find systematically confused classes.

Rubric key points.

• Guideline, calibration batch and a versioned edge-case list before volume labeling.
• Gold set built first, adjudicated, random, frozen as the test set.
• Cost per usable label (0.375 crowd, USD 1.00 on the whole program) is higher than the sticker price; arithmetic shown correctly.
• Honeypot power argued with a binomial calculation, not asserted.
• Test-set noise (std 0.030 at 500) justifies the gold size; rare-class counts are mentioned.
• Provenance and guideline versioning stored with every label.