Tokens and tokenization

A token is the unit of text a language model actually reads and writes. It is usually a bit smaller than a word: common words are one token, rarer words are cut into a few pieces. Tokenization is the fixed, reversible procedure that chops any text into these units before the model sees it, and stitches them back together when the model is done.

The model needs numbers, so text has to be cut up

A language model is, underneath, a large arithmetic machine. It multiplies and adds numbers. It cannot operate on letters or words directly, because those are not numbers. So the first thing that has to happen to any text, before the model does anything at all, is that the text gets converted into a sequence of numbers the machine can work with. The pieces you cut the text into are the tokens, and each distinct token has a fixed number, its id, in a big list called the vocabulary.

Think of the vocabulary as a numbered phone book of text fragments. Every fragment the model knows sits at a fixed position. Tokenizing a sentence means looking each fragment up and writing down its number. Turning the model’s output back into readable text means doing the reverse lookup, number back to fragment, and gluing the fragments together. The process runs in both directions with nothing lost, which is what “reversible” means here: the exact original text, spaces and punctuation and all, can always be recovered from the token numbers.

Why not just use whole words

The obvious idea is to make every word a token. It fails on contact with real language. The first problem is size. If every distinct word needs its own slot in the vocabulary, you need a slot for every word in every language you support, plus every name, every typo, every made up product name, every technical term. That runs to millions of entries and still misses things. The second problem is worse: any word not in the list has no representation at all. The model would meet a new word, find no number for it, and be stuck. New words appear constantly, so a word based vocabulary is out of date the moment it is built.

There is also a subtler waste. Treating “run,” “runs,” “running,” and “runner” as four unrelated entries throws away the obvious fact that they share a root. The model would have to learn each one separately from scratch, as if they had nothing to do with each other. A good unit should let the model reuse what it learns about “run” across all of them.

Why not go all the way down to characters

The opposite idea fixes those problems and creates a new one. Make every single character a token: the 26 letters, the digits, the punctuation, the space. Now the vocabulary is tiny and nothing is ever out of the list, because any text is just a sequence of characters. But you have thrown away all the structure. The word “understanding” becomes thirteen separate tokens with no hint that they form one word, and the model has to rebuild the very idea of a word from letters every single time. Worse, sequences get extremely long. A page of text that would be a few hundred word sized units becomes a couple of thousand character sized ones, and as a later idea explains, longer sequences cost more to process and eat into a fixed budget.

The middle path: subword tokens

The approach that actually gets used sits between the two extremes and takes the best of each. It is called subword tokenization. The rule of thumb is simple to state: frequent chunks of text get their own token, and infrequent chunks are built out of smaller pieces. The vocabulary is not chosen by hand. It is learned from a large sample of text by a procedure that starts from individual characters and repeatedly merges the pair of pieces that appears together most often, keeping the merges that pay off, until it has a vocabulary of a fixed size, often somewhere around one hundred thousand entries.

What comes out of that procedure lines up with intuition. Because “the” and “and” and “of” appear constantly, each becomes a single token. Because common word parts like “ing” and “tion” and “un” appear constantly, they become tokens too. Rare and novel words, which by definition do not appear often enough to earn their own slot, get assembled from these smaller common pieces. The result is a vocabulary that is compact, never gets stuck on an unknown word, and still keeps enough structure that related words share pieces.

Common text is cheap in tokens and rare text is expensive, because the tokenizer spends whole tokens on what it sees often and spare change on what it sees rarely.

Watching a word come apart

A concrete example makes the whole idea click. Take a short, ordinary sentence. “The cat sat.” A typical tokenizer turns that into four tokens: a token for “The,” a token for “ cat” with its leading space, a token for “ sat” with its leading space, and a token for the period. Three common words and a punctuation mark, four tokens, roughly one per word. This is the easy case and it is the common case, which is why people often say a token is about three quarters of a word on average.

Now take a word the tokenizer has rarely or never seen as a whole, like a technical term or an unusual name. Something like “tokenization” itself may come apart into a few pieces, for example “token,” then “ization,” so one word becomes two tokens. A stranger string, like a random identifier “xq7-plumbus,” might shatter into many small pieces, a token or two per chunk, because none of it matches a common entry. The pattern is consistent: the more familiar the text, the fewer tokens it takes; the more unusual, the more it fragments. The reassembly is exact in every case, since the pieces still carry their leading spaces and combine back into the original characters.

How code and whitespace tokenize

Source code goes through the same machine, and it behaves in ways worth knowing if you feed code to a model. Keywords and common symbols that appear all over code, like “return,” “def,” “function,” an equals sign, an open parenthesis, tend to be single tokens because they are so frequent. But identifiers that a programmer invented, like a variable named “userAccountId,” often split into several pieces, since the tokenizer has no single slot for that exact name. A name written as “user_account_id” and one written as “userAccountId” can tokenize into different numbers of pieces even though they mean the same thing to a person, because the underscore and the capital letters fall at different boundaries.

Whitespace is the detail people miss. Spaces and newlines are not free. Many tokenizers attach a leading space to the token after it, so the same word costs a token whether or not it has a space in front. Runs of spaces, the kind that indentation produces, can each cost tokens. Deeply nested code, indented several levels, spends real tokens on the indentation alone. Here is the shape of it, where each visible unit is roughly what becomes a token.

def add(a, b):
    return a + b

In that tiny snippet, “def,” “ add,” the parentheses, “a,” the comma, “ b,” the colon, the newline, the four spaces of indentation, “return,” and each part of “a + b” all cost tokens. A two line function is easily a dozen or more tokens. Multiply that across a large file and it adds up quickly, which is exactly why the amount of code you can hand a model at once is more limited than the number of lines would suggest.

Every model brings its own scissors

One point saves a lot of confusion: there is no single, universal way to cut text into tokens. Each model is trained alongside its own tokenizer, with its own vocabulary learned from its own sample of text, so the same sentence can become a different number of tokens depending on which model you ask. A word that is a single token to one model may be two to another, because the first happened to see it often enough during vocabulary building and the second did not. This is why a token count is only exact for a specific model, and why a rule of thumb like “a token is about three quarters of a word” is a rough average across ordinary English rather than a fixed conversion.

The differences are widest away from plain English. A tokenizer built mostly on English text will spend more tokens on another human language, or on unusual symbols, because those pieces were rarer in its training and did not earn compact slots. The same handful of sentences can therefore be noticeably cheaper for one model than another, purely because of how each one’s scissors were shaped. None of this changes the meaning of your text; it only changes the count, and the count is what you pay for and what has to fit.

Why token counts decide cost and limits

Tokens are not just an internal detail. They are the unit almost everything about using a model is measured and priced in, so it pays to think in them. Two facts follow from that, and both matter in practice.

First, cost. Model providers charge by the token, both for the text you send in and the text the model sends back. A request is not billed by the word or the character or the question. It is billed by how many tokens the tokenizer produced. Two prompts that read as the same length to a person can cost different amounts if one is full of common words and the other is full of rare names, code, or unusual formatting that fragments into more tokens.

Second, limits. Every model can only consider a fixed number of tokens at once, its context window, which the next idea covers in full. The relevant point here is that the window is measured in tokens, not in words or files. Whether a given document fits depends entirely on how many tokens it becomes, and that is exactly why prose and code and dense identifiers pack so differently. A file that looks modest can consume a surprising share of the window once the indentation and the invented names are counted.

The same idea, different token costs

Here is a point that ties the whole thing together and often catches people out. The token count depends on the surface form of the text, not on the meaning. The same idea can cost very different numbers of tokens depending on how it is written. A single, familiar English word for a concept may be one token, while a precise technical phrase for the same concept may be five. An acronym everyone uses may be a single token, while its spelled out expansion is several. Writing a number as “1000000” tokenizes differently from writing it as “one million.”

None of this changes what you meant. It only changes how the tokenizer cut it and therefore what it costs and how much of it fits. This is a useful lever: phrasing things in common, plain language, and avoiding needless formatting or exotic characters, genuinely reduces token count. It is also a useful warning, since content that looks small on screen, like a wall of deeply nested code or a table of odd identifiers, can be surprisingly heavy in tokens.

Tokens, then, are the raw material everything else is built from: the units the model reads, the units you are billed in, and the units the model’s fixed budget is measured in. Two ideas follow directly. One is that fixed budget, the amount of token history a model can hold in mind at once, which is the subject of the context window. The other is a deeper question about what a token even means to the model once it is inside: how a bare token id gets turned into something that carries meaning, which is the subject of embeddings and vector space.