Design a Classifier for 40 Imbalanced Support Classes

1. Splitting. Deduplicate first (exact and near-duplicate, for example MinHash on word 3-grams), then split by duplicate group, then by time: train on the older months, validate on the next, test on the most recent. A random row split leaks re-sent and forwarded tickets: in the lesson's synthetic inbox (21.4 percent duplicate rows) a random split gave macro-F1 0.763 for a linear SVM while the group split gave 0.707 and the time split 0.696; the more a model can memorize the larger the gap (1-nearest-neighbour: 0.676 versus 0.545). Time order also exposes vocabulary drift from new features, which is exactly what production sees. Keep a gold set labelled carefully by two annotators, because the agents' labels are themselves noisy (about 8 percent disagreement means a ceiling near 92 percent agreement before any model error).

2. First model. TF-IDF on word 1-2-grams (min_df of 2, sublinear tf) plus character n-grams if tickets contain typos, then a linear SVM or logistic regression. Fit the vectorizer inside the pipeline on training data only. This trains in about a second on 7,000 tickets and predicts about 58,000 tickets per second on one thread in my measurement, so you can iterate quickly. Lead with macro-F1 and per-class precision and recall, plus the confusion matrix, because 22 percent of the volume sits in one queue and accuracy (0.748 versus macro-F1 0.664 in the lesson's run, and 0.221 versus 0.013 for always predicting the top class) will flatter it. Keep accuracy and weighted-F1 as secondary numbers that track business volume.

3. The rare queue. With about 600 tickets a month you have thousands of examples over 18 months, so first try class_weight="balanced" or a per-class threshold. Be honest about what it does: in the lesson it moved macro-F1 by +0.002 for logistic regression and -0.012 for the SVM, so it is a trade-off to test, not a fix. Check whether the queue is a taxonomy problem (overlaps with a neighbour; agents disagree) before modelling harder: error analysis on 100 errors often shows the biggest bucket is ambiguity. If the class really is separable but has few examples, add labels through active learning or an LLM-assisted labeling pass with a human audit (Data Labeling, Active Learning & Weak Supervision).

4. Auto-routing. Calibrate the probabilities on a validation set (the same data at two regularization settings had ECE 0.040 and 0.177), then choose a confidence threshold per queue, or a global one, so that auto-routed tickets reach a target precision (say 95 percent) and report coverage at that precision. Everything below the threshold goes to humans. Log the model version, confidence and the agent's final label; that stream measures live accuracy and becomes the next training set (ML Monitoring, Drift & Retraining). Add an "other / unsure" path and never auto-route tickets in a high-risk class.

5. If the baseline is not enough. Decide from the error analysis. If errors are paraphrases, negation or context that keywords miss, try a fine-tuned encoder (a DeBERTa or RoBERTa class model with learning-rate warmup, 2 to 4 epochs, several seeds); if tickets are long, choose head+tail truncation or sliding windows according to where the evidence sits. If errors are label noise or class overlap, an encoder will not fix them: fix the guidelines or merge classes. If the taxonomy changes weekly, use an LLM to label new classes and distil into the small model. In every case compare on the same group-and-time split against the TF-IDF floor, and price serving as well as accuracy.