How Consistently BPE Tokenizes Multi-Digit Numbers
Your team is estimating token costs for a financial dashboard that
generates lots of numeric output — prices, IDs, percentages. A
teammate proposes a simplifying assumption: "Let's just budget one
token per digit, so a 6-digit number like 420000 costs 6 tokens,
consistently." Based on how BPE tokenizers actually handle numbers,
is this a safe assumption?
A. Yes — BPE vocabularies include a dedicated single-digit token for every digit 0-9, so digit count and token count are always equal. B. No — BPE has no clean one-token-per-digit rule; digit grouping falls out of ordinary merge-table statistics, so token counts per number are inconsistent and can't be predicted from digit count alone. C. No — numbers are always chunked into fixed groups of three digits (thousands, like comma-formatted output), so token count is digit count divided by three, rounded up. D. Yes, but only for numbers under five digits; every number of five or more digits is always exactly two tokens.
Share this question