Evaluate an NER Model: Token Accuracy Versus Entity F1

1. Token accuracy. Compare tag by tag: the first seven tokens ("Dr" through "Star") match, "Bank" is gold I-ORG but predicted O (wrong), "in" matches, "Madrid" is gold B-LOC but predicted B-ORG (wrong). That is 8 of 10 correct, so accuracy = 0.80. Only four of the ten gold tags are O, so an all-O predictor would score 0.40 here; on realistic text the O share is often 80 to 95 percent, so an all-O model can score that high in accuracy while finding nothing.

2. Token-level non-O metrics. Predicted non-O tokens: Maria, Lopez, North, Star, Madrid = 5. Of these, 4 have the gold tag (Maria, Lopez, North, Star); Madrid has the wrong type. Gold non-O tokens: Maria, Lopez, North, Star, Bank, Madrid = 6. Precision = 4/5 = 0.80, recall = 4/6 = 0.667, F1 = 0.727.

3. Entity-level strict scoring. Gold entities: PER (tokens 1 to 2), ORG (tokens 5 to 7), LOC (token 9). Predicted entities: PER (1 to 2), ORG (5 to 6), ORG (9). Only the PER span matches exactly in both boundary and type. The ORG prediction is a boundary error (it dropped "Bank"), so it is both a false positive and, for the gold ORG, a false negative; the Madrid prediction has the wrong type, again one FP and one FN. So TP = 1, FP = 2, FN = 2, precision = 1/3, recall = 1/3, F1 = 0.333. (I checked these with a hand-written conlleval-style scorer.)

4. Why they differ and what to report. Token accuracy is dominated by the easy O tokens. Token F1 gives partial credit to a span that is three-quarters right, but downstream systems consume whole entities: a truncated "North Star" cannot be linked to "North Star Bank", and a LOC mislabelled ORG breaks routing. A boundary error also costs twice at entity level (FP and FN). Report strict entity-level precision, recall and F1, overall and per type, say which tagging scheme and which strictness mode you used (strict versus lenient, how a stray I- is handled), and optionally add a lenient or partial-match score as a diagnostic. In the lesson's six-sentence toy set the same gap appeared: token accuracy 0.806, token F1 0.743, strict entity F1 0.600.

5. Other pitfalls.

• Subword alignment: with a transformer tokenizer a word becomes several subtokens; if you score subtokens (or count the continuation pieces) the numbers differ from word-level scoring. Train on the first subtoken and mask the rest, predict per word, and evaluate at word or character-span level.
• Illegal tag sequences: O I-PER or B-PER I-ORG come from independent softmax tagging; decide whether to repair them (conlleval-style does) or count them as errors, or add a CRF layer or constrained decoding.
• PII and recall: for PII the cost is asymmetric. A missed phone number or card is worse than a redacted non-secret, so report entity-level recall per PII type at the operating threshold, test on realistic messy text, and combine rules with validators (for example the Luhn check for card numbers).
• Nested entities and overlapping annotations: flat BIO cannot represent them, so scores depend on a convention (outermost only, or span-based scoring); say which.
• Annotation consistency: if two annotators disagree on whether "North Star Bank" or "North Star" is the span, a model cannot score above that agreement; measure agreement (span-level F1 between annotators) first.
• Domain shift: a model tuned on news has poor recall on tickets; evaluate on in-domain data and per entity type, not on the CoNLL average.