Tokenization and Context Windows

Every number an AI Engineer quotes in an interview — cost per request, latency, "how much history can we keep", "will this fit" — is denominated in tokens, not words or characters. Get the mental model of tokenization wrong and every downstream estimate is wrong with it: a "200k context window" model that "should easily fit our whole codebase" turns out to choke on a directory of minified JSON; a cost estimate based on word count is off by 30–50%; a prompt that technically fits the window degrades silently because the one fact that matters got buried in the middle of it. None of this is trivia. It is the layer underneath every other AI-engineering decision — RAG, agents, fine-tuning, cost modeling — and interviewers probe it precisely because candidates who have only used chat UIs have never had to think about it.

This subject treats tokenization and context budgeting as a single topic because they are the same constraint viewed from two angles: tokenization determines how much of your text becomes billable, latency-costing units, and context budgeting is the discipline of deciding which tokens earn a place in the window given that a bigger window is not free and — past a point — not even fully effective. llm-application-system-design introduces a token budget table in passing; this subject is the standalone deep dive, and rag-architecture-end-to-end picks up where it leaves off, covering how retrieval turns an unbounded corpus into a token budget you can actually afford.

The practical payoff: after this subject you should be able to look at a piece of text and estimate its token count within ~20% without a tokenizer in front of you, explain why a model with a 200k window still needs retrieval, and build a token budget table for a real prompt the way you'd build a memory budget for an embedded system — line by line, with a total, and with levers named for when it doesn't fit.