What a BPE Tokenizer Does With a Word It Never Saw
A BPE tokenizer's vocabulary was built from a training corpus that never contained the word "hyperparameterization" as a whole unit. At inference time, a user's prompt includes that exact word. Which of the following best describes what the tokenizer actually does with it?
A. It falls back to a single special <unk> (unknown) token, since
the word was never seen during vocabulary training.
B. It greedily applies the merge table it learned during training,
breaking the word down into whichever known subword pieces the
merges produce (e.g., something like "hyper", "parameter",
"ization").
C. Tokenization fails and the API request returns an error, since
BPE vocabularies cannot represent out-of-vocabulary words.
D. It matches the word to the closest whole word already in the
vocabulary by edit distance and substitutes that instead.
Correct answer: B — "It greedily applies the merge table it learned during training, breaking the word down into whichever known subword pieces the merges produce."
BPE vocabularies are built bottom-up from individual bytes/characters, with merges learned as an ordered list of "these two symbols combine into this one" rules. Because the base vocabulary starts at the byte/character level, any string is representable — tokenizing new text just means starting from bytes/characters and greedily applying the highest-priority applicable merge until none apply. A word never seen whole during training, like "hyperparameterization", still decomposes cleanly into subword fragments the vocabulary does know, the same mechanism shown in the worked "lowest" example in this subject. This is precisely what lets BPE generalize to unseen words, typos, code identifiers, and other languages without ever needing an escape hatch.
Why the distractors are wrong
- A describes how older, closed-vocabulary word-level tokenizers
handled out-of-vocabulary words. Subword BPE tokenizers are
specifically designed so an
<unk>-style fallback is essentially never needed for representable text — that's the whole point of building the vocabulary from bytes/characters up. - C is wrong for the same reason: nothing about BPE's design requires whole-word membership in the vocabulary, so there's no failure mode where an unseen word causes an error.
- D invents a nearest-neighbor substitution mechanism that isn't part of how BPE tokenizes; substituting a different word would also silently corrupt the model's input, which is not what happens — the original characters are always preserved, just split into more pieces.
Share this question