Practice — Classical NLP & Text Classification (6 questions)

A support inbox receives 300,000 tickets a month, routed by agents into 40 queues. The largest queue holds 22 percent of tickets and the smallest 0.2 percent (about 600 per month). You have 18 months of history, an agent-assigned label on every ticket (agents disagree on roughly one ticket in twelve), and a product that adds or renames a feature every few weeks. Management wants "auto-route where it is safe, and send the rest to humans".

Describe the design you would build and defend:

1. How you split the data, and why a random split is wrong here.
2. The first model, its features, and the metric you lead with.
3. What you do about the 0.2 percent queue.
4. How you decide which tickets are auto-routed.
5. What you would try second if the baseline is not good enough, and what evidence would tell you to try it.

An NER model is evaluated on one sentence in BIO tags. Tokens: Dr Maria Lopez works at North Star Bank in Madrid.

• Gold: O B-PER I-PER O O B-ORG I-ORG I-ORG O B-LOC
• Predicted: O B-PER I-PER O O B-ORG I-ORG O O B-ORG

1. Compute token accuracy.
2. Compute token-level precision, recall and F1 over non-O tags (a token counts as correct only if the predicted tag equals the gold tag).
3. Compute strict entity-level precision, recall and F1 (span and type must both match).
4. Explain why the numbers differ, and what you would report to a stakeholder.
5. Name four other pitfalls when evaluating NER, including the ones that arise with transformer subword tokenizers and PII use cases.