Data Labeling, Active Learning & Weak Supervision

Getting the labels that set your model's ceiling — cheaply, measurably and without poisoning your eval set

Learn to design a labeling program with numbers you can reproduce, on a synthetic 100,000-message support inbox with ten imbalanced classes and a 2,000-label budget: why labels set the ceiling (a 20 percent symmetric flip rate cost 0.083 macro-F1 and a class-conditional one 0.127, and a noisy test set misreports even a clean model); agreement statistics implemented by hand and checked against a hand-computed example (Cohen kappa 0.4, Fleiss kappa 1/3, Krippendorff alpha 7/18) with annotators of 85, 80 and 75 percent accuracy reaching only kappa 0.56 to 0.65, honeypot detection power, and a budget in which a USD 0.30 sticker price became USD 1.00 per usable label; pool-based active learning where margin, entropy, uncertainty and a margin-plus-k-means hybrid reached random sampling's 2,000-label macro-F1 with roughly half the labels while pure k-center diversity lost to random, uncertainty sampling bought 35 percent no-signal messages, and evaluating on the AL-selected slice read accuracy 0.432 against 0.664 on a random test set; weak supervision with twelve labeling functions, coverage, precision and conflict tables, majority vote against a Dawid-Skene style label model fitted by EM (similar accuracy, 0.522 versus 0.641 macro-F1), and a correlated-LF failure where four copies of a weak rule drove label-model confidence to 1.000 at 0.218 accuracy; pseudo-labelling and confirmation bias (every threshold failed to beat the supervised baseline here, and rare-class recall fell from 0.160 to 0.033); an explicitly assumption-based LLM-labeling cost table, a simulated distillation and the LLM-labeled-test-set hazard; synthetic data and a measured toy model-collapse demonstration; confident learning by hand with detection precision and recall for injected errors; and versioning, lineage and datasheets. Every measured number comes from fixed-seed scripts on synthetic data; prices and LLM accuracies are labelled placeholders.

Practice questions (6)

• Design a Labeling Program for 100,000 Messages on a Fixed Budget

  Intermediate · Free
  View →

• Choose an Active Learning Strategy for an Imbalanced Ten-Class Problem

  Intermediate · Free
  View →

• Write Three Labeling Functions and Analyse Their Conflicts

  Intermediate
  View →

• Validate LLM-Produced Labels Before Using Them

  Intermediate
  View →

• Find Mislabeled Examples in an 8,000-Item Training Set

  Advanced
  View →

Quizzes (1)