Practice — Data Labeling, Active Learning & Weak Supervision (6 questions)

A support team has 100,000 unlabeled customer messages and ten classes (the rarest is 2 percent of traffic). The budget is USD 3,000 in total. Placeholder prices for this exercise: a crowd label costs USD 0.30, an in-house expert label USD 1.20 and an adjudication of a disputed item USD 2.00. Two in-house annotators of 0.80 accuracy, with errors spread uniformly over the nine wrong classes, disagree on 35.6 percent of items.

1. Design the program: what you build first, who labels what, and how you detect bad annotators.
2. Do the budget arithmetic for a 300-item gold set labeled by three in-house annotators and then for a 500-item gold set. How many crowd-labeled training items remain in each case if quality control re-labels 10 percent of them twice more and 5 percent extra items are honeypots?
3. Say which gold set size you choose and why, using the noise of a small test set.
4. State what you would measure to know the program is working.

You have a pool of 100,000 unlabeled messages in ten imbalanced classes (the three rarest are 3.9, 3.0 and 2.0 percent of traffic), a logistic-regression text classifier and a budget of 2,000 labels in rounds of 200. A colleague proposes pure uncertainty sampling; another proposes k-center diversity sampling; a third says random sampling is enough.

1. Work out how many items of the three rarest classes random sampling is expected to collect in 2,000 labels.
2. Choose a strategy and state the cold-start plan, the batch mode and the stopping rule.
3. Explain how you evaluate the strategy so the result is not biased by the selection.
4. Give the failure modes of your choice and how you would detect each.