Nib asked for
Chopping Text into Tokens
Before the model can work with your words, it splits them into tokens, chunks of text that are often whole words but sometimes fragments, and turns each token into a number.
5 steps, drawn by Nib. Read it here, or watch him draw it. Updated .
1what's a token
The last page said ChatGPT predicts the next piece of text. But what is a 'piece'? The model can't work with letters or sentences as you see them. Its first job is to chop your message into small chunks called tokens.
The model's 'pieces' of text are called tokens.
2whole or split
Common short words like 'the' or 'cat' usually stay whole, one token each. Longer or rarer words get broken into familiar fragments, so 'unbelievably' might become 'un', 'believ', 'ably'. A space or punctuation mark often sticks to the chunk beside it. The exact splits vary from model to model.
Common words stay whole; rare words split into fragments.
a common mix-upMany people assume the model reads whole words, like a person does. It actually works on chunks that only sometimes match words.
A word the model has never seen before, like a brand-new product name, arrives. Can it still be turned into tokens?
Show the answer
Yes. It gets split into smaller familiar fragments, down to single letters if needed, so nothing is ever unreadable.
3Lego bricks
Why fragments at all? A list with one entry for every word in every language, plus every name, typo, and code snippet, would be endless. A list of a few tens of thousands of reusable chunks can build almost any text, the way a small set of Lego bricks builds almost any shape.
A fixed menu of reusable chunks can build any text.
4chunks to numbers
Next, each token in that menu has a number, its ID. 'cat' might be 9246. The tokenizer swaps every chunk of your message for its number, so your sentence becomes a list of numbers. From here on, the machine never sees letters, only numbers.
Each token is replaced by its numeric ID.
Once your message is a list of numbers, does the model still have access to the individual letters of your words?
Show the answer
Not directly. It only has the token numbers, which is why letter-level tasks, like counting the letters in a word, can trip it up.
5in, then out
Once the whole message is turned into numbers, the model takes in all of those tokens together. Then it produces its reply one token at a time, each new token added to the pile before the next guess. That is the same guess-and-add loop from page 1, now with a clear definition of what it is guessing: the next token's number.
It takes in all tokens together, then replies one token at a time.
So your words become chunks, and chunks become numbers. Everything the machine does next, learning, weighing, guessing, is done on those numbers.