Choose an Active Learning Strategy for an Imbalanced Ten-Class Problem

1. Expected rare items under random. The three rarest classes are 3.9 + 3.0 + 2.0 = 8.9 percent of traffic, so 2,000 random labels contain 2{,}000 \times 0.089 \approx 178 of them, about 78 for the 3.9 percent class, 60 for the 3.0 percent class and 40 for the 2.0 percent class. A classifier trained on 40 examples of a class has poor recall for it, which is why macro-F1 is the metric to watch, not accuracy.

2. A choice that fits the evidence. In the worked example (5 seeds, 200 random initial labels, the same start for every strategy) the macro-F1 at 2,000 labels was random 0.530, uncertainty 0.622, margin 0.632, entropy 0.622, k-center 0.452 and a hybrid (top 2,000 by margin, then one item per k-means cluster) 0.639. I would choose margin sampling in hybrid batch mode, with this plan:

• Cold start: 200 to 400 random (or keyword-seeded) labels so every class appears; uncertainty from a model that has seen 200 labels is close to meaningless (all strategies scored 0.152 at 200).
• Batches: 200 per round, take the 2,000 most uncertain by margin and pick one per k-means cluster on the embeddings to avoid near-duplicates; keep 10 to 20 percent of every batch random.
• Review the queue: hand-inspect the top of each batch and tag unlabelable items "unsure"; uncertainty sampling spent 35 percent (least confidence and entropy) and 25 to 28 percent (margin and the hybrid) of its picks on messages with no signal at all, against a 12 percent base rate.
• Stopping: stop when the marginal gain per batch falls below what a batch is worth (for margin the last two rounds added +0.017 and +0.011 macro-F1 per 200 labels), when predictions on a fixed probe set stop changing, or when the budget ends.

The gain is real but not free of caveats. The uncertainty-like strategies collected 297 to 390 items of the three rarest classes against 181 for random, so a large part of the macro-F1 gain is rare-class discovery; the accuracy of uncertainty and entropy sampling (0.660, 0.659) was no better than random's 0.667. A class-stratified random baseline was not tested and may close part of the gap. K-center was worse than random (-0.078 macro-F1), so diversity alone is not a safe default.

3. Evaluation. Keep a random test set drawn before any selection and never trained on, and compare strategies at equal label counts over several seeds with paired differences. Do not hold out part of the selected set: training on 1,500 margin-selected items and testing on the other 500 gave accuracy 0.432, while the true random test accuracy was 0.664 (uncertainty: 0.382 against 0.632). The selected items are the hard ones, so the slice misreports the model by 0.23 to 0.25; in other problems the bias goes the other way.

4. Failure modes and detection.

• Irreducible noise selected: measure the share of picks that annotators mark "unsure" or that get three different labels.
• Redundancy: measure the mean pairwise embedding similarity inside each batch.
• Sampling bias in estimates: compare the class mix of the labeled set with the pool and reweight or use the random share for estimates.
• Model dependence: re-test the labeled set with a different model family before trusting it as a general-purpose training set.
• Undiscovered classes: uncertainty finds classes the model already half-knows, so a periodic random or novelty batch is the only guard.

Rubric key points.

• Correct expected rare-class count (about 178) and the point that this makes macro-F1 the metric to watch.
• Cold start with random labels, a hybrid or margin strategy with diversity inside the batch, and a random share.
• Evaluation on an untouched random test set, with the selection-bias numbers or the argument for them.
• Honest statement that k-center lost to random here and that gains were largely rare-class discovery.
• A stopping rule based on marginal gain per batch.