Data Labeling, Active Learning & Weak Supervision

This lesson is about where labels come from, how good they are, and how to spend a small budget on the ones that matter. The boundary: preference data for aligning LLMs (pairwise comparisons, reward models, annotator calibration for RLHF) is Case Study: Designing an RLHF / Preference-Tuning Platform; this lesson is the general supervised-learning labeling program (classification, extraction, ranking relevance) and the techniques that cut its cost. Metric theory is Model Evaluation Metrics; the text models used in the examples are Classical NLP & Text Classification; the pair-labeling instance of active learning is in Entity Resolution & Record Linkage; drift and relabeling triggers are ML Monitoring, Drift & Retraining.

Interviews on this topic ask "you have 100,000 unlabeled items and budget for 2,000 labels, what do you do?" Strong answers cover five decisions: what the guideline says, how you measure that humans agree, which items to label, which labels to get cheaper than a human, and how you keep the evaluation set clean. Every number marked "measured" comes from scripts in /tmp/lab/ run on a synthetic support-message inbox (templated text with typos and noise, fixed seeds, described in the worked example). The data is synthetic and cleaner in structure than any real inbox, so read the numbers as mechanisms and orders of magnitude, not benchmarks. Prices and LLM accuracies are labelled placeholders, and anything from a paper is recalled from memory.

Labels as the Ceiling

A model learns the labeling process, not the world. If two careful people disagree on 20 percent of items, no model is going to be "more right" than that on the same definition, and any noise in training labels lowers what a model can reach with the same data. Four things set the ceiling:

• The task definition. "Is this message about billing?" hides decisions: does a refund request about a billing error count? Ambiguous classes produce disagreement that no amount of labeling money removes. Fix the taxonomy first (merge classes annotators cannot separate; split classes that hide two behaviours).
• The guideline. One page of definitions, positive and negative examples per class, and a list of edge cases with the decision made. Edge-case documentation is the highest-value artifact: each disagreement found in calibration becomes a rule, with a date and an owner. A guideline is versioned like code (see Versioning).
• The task design. Closed choice beats free text. Multi-class forms with more than about 10 options degrade; use a hierarchy (coarse then fine) or a "none of these / unsure" escape so annotators do not guess. Show context the model will see and nothing it will not; show pre-filled model suggestions only knowing it anchors annotators (faster, but agreement with the model rises for the wrong reason).
• The tooling. Keyboard shortcuts, keeping the queue short, skipping ("cannot decide") as a first-class label, per-item time tracking and a way to flag guideline gaps. Throughput depends more on the tool than on the hourly rate.

How much does noise cost? In the worked example, a logistic regression trained on 8,000 messages reached test macro-F1 0.710 with clean labels. Flipping labels uniformly to another class at 10, 20 and 30 percent gave 0.668, 0.627 and 0.585; flipping to a confusable partner class (refund to billing, privacy to login, and so on) gave 0.659, 0.583 and 0.528. Class-conditional noise is worse because it is systematic: the model has no way to average it out. Noise in the test labels is a separate cost: the same model, scored against test labels flipped at 20 percent, reads 0.469 instead of 0.627. A cleaner test set is therefore worth more per label than a cleaner training set, a point that returns in the budget section.