Nib asked for
Where Tokens Come From
Before training, text is chopped into tokens (common chunks of characters and whole words) by a fixed procedure, so one word can become one token or several.
5 steps, drawn by Nib. Read it here, or watch him draw it. Updated .
1who chops?
Page 1 said the model receives numbers standing for pieces of text. But who decides what a 'piece' is? Not the model, and not a person reading each word. The pieces are fixed ahead of time, before the model learns anything, and they are called tokens.
Tokens are fixed pieces of text, chosen before the model ever trains.
2merge pairs
The procedure starts by treating text as individual characters. It scans a huge pile of text and finds which neighboring pair of pieces shows up together most often. It glues that pair into one new piece, then scans again. Repeating this thousands of times builds up a vocabulary of useful chunks.
Repeatedly gluing the most frequent neighboring pair builds the chunk vocabulary.
3fusing up
Picture this on s-t-r-a-w-b-e-r-r-y. Imagine that, in the training text, the pair 'e' and 'r' appears together very often, so the procedure merges them into 'er'. Later rounds might glue 'st' and 'str' and eventually whole chunks like 'straw'. The exact merges depend on the text it scanned; this is an example of the mechanism, not a prediction for any particular model.
Each merge turns a frequent pair into one piece, and merges stack into bigger chunks.
a common mix-upThe merged pairs are not chosen by looking at spelling rules or meaning; they are chosen only by how often characters sit together in the training text.
4the vocabulary
The result is a fixed vocabulary, often tens of thousands of entries, each with its own number. Very common words like 'the' or 'water' end up as a single token. Rare or long words get split into several smaller tokens. Words that were never merged fall back to smaller chunks or even single characters, so any text can still be written.
Common words become one token; rare words split into several.
5only numbers
Once the vocabulary is fixed, typing a word means looking up its chunks and swapping each for its number. Notice what this means for letters: a chunk is one indivisible entry in the list. The model gets the number, not the characters inside it. That sets up the next page's question about strawberry.
Each chunk becomes one number; the characters inside it are not passed along.
If a word is common in the training text, will it more likely be one token or several?
Show the answer
More likely one token, because frequent pieces get merged into single vocabulary entries.
So a word's pieces are decided by frequency in old text, not by its letters, and next we'll see what that does to strawberry.