Nib asked, and Nib — askNib's tutor — drew the answer live at a whiteboard. This is the spoken transcript; enable JavaScript to watch it drawn.
How do AI language models like ChatGPT actually work? (for a curious adult who uses chatbots but has never studied computing). Cover, in this order: How does ChatGPT work?; How does it read your words?; How does it learn which words come next?; How does it know which earlier words matter?; Why do AI chatbots hallucinate?; Why does it forget what you said earlier?; How did a text predictor learn to follow instructions?; Does it understand, and how much should you trust it? — page 2: Chopping Text into Tokens
Last page, the machine guessed the next piece of text. But what is a piece? It is a token: a chunk of text, sometimes a word, sometimes part of one.
Everyday words stay whole. A long or rare word gets broken into familiar fragments, like un, believ, ably. Punctuation and spaces join in too.
Storing every possible word would never end. Instead, keep a fixed menu of reusable pieces, like Lego bricks, and snap them together to build any word.
Every chunk on the menu has its own number. Replace each chunk in your message with its number, and the sentence becomes a list of numbers.
The model takes in all your tokens together, then writes its reply one token at a time, each new one added before the next guess.
So your words become chunks, and chunks become numbers. Everything the machine does next, learning, weighing, guessing, is done on those numbers.