Nib asked for
Learning From Mountains of Text
The model learns by guessing the next token over and over across enormous amounts of text, and after each guess it nudges its billions of internal numbers slightly toward whatever would have made the right answer more likely.
6 steps, drawn by Nib. Read it here, or watch him draw it. Updated .
1mountains of text
Page 1 said the model predicts a likely next piece of text, and page 2 said text becomes numbered tokens. Now the question is where good predictions come from. Nobody writes rules for them. Instead the model is trained by reading enormous amounts of text: books, articles, websites, code.
Good guesses aren't programmed in; they're learned from text.
a common mix-upPeople often picture engineers typing in grammar rules and facts by hand.
2guess, then check
Training turns ordinary text into a quiz that answers itself. Take a sentence, hide its next token, and ask the model to guess. The real text supplies the right answer for free, so no human has to grade anything. Every sentence in the pile becomes thousands of such quizzes.
Hiding the next token turns any text into a self-grading quiz.
Why doesn't anyone need to label the training text by hand?
Show the answer
The text itself contains the correct next token for every position, so the answer key is built in.
3random knobs
The model holds billions of adjustable numbers called parameters. Think of them as knobs on a gigantic mixing board. Early in training they are set randomly, so the first guesses are nonsense, close to picking tokens at random. All of the model's eventual skill will live in how these knobs end up set.
Parameters are billions of adjustable knobs, and they start out random.
a common mix-upThe parameters are not stored sentences or facts; they are just numbers that shape each guess.
4tiny nudge
After each guess, the model is compared with the real answer. If it gave the right token only a small chance, every knob gets a tiny push in the direction that would have raised that chance. The nudge is slight on purpose: one sentence should not rewrite the model, it should only lean it a little.
Each wrong guess nudges every knob slightly toward the right answer.
Why is each nudge kept very small?
Show the answer
So no single sentence overwrites what earlier text taught; learning comes from many small pushes adding up.
5skill climbs
Now repeat this an astronomical number of times, over text drawn from a large share of the written internet. Early nudges fix spelling-level patterns, later ones capture grammar, then facts, styles, and ways of reasoning, because all of those help predict what comes next. No one tells the model which patterns to find; whatever reduces its mistakes gets baked into the knobs.
Billions of tiny nudges add up to grammar, facts, and style.
6what remains
When training ends, the knobs are frozen and the text is gone; only the settings remain. This is why the model can write about things it never saw word for word: it learned patterns, not a copy. It is also why it can be wrong, since patterns learned from text are not a verified database. That worry belongs to a later page.
What remains after training is a pattern of settings, not a library of the text.
a common mix-upThe model does not look things up in stored documents when it answers.
So a chatbot's skill is not programmed or looked up: it is the leftover shape of billions of tiny nudges from guessing, checking, and adjusting, over and over.