Nib asked for
A Machine That Guesses Next
ChatGPT writes by repeatedly guessing a likely next piece of text, adding it to the text so far, and guessing again.
5 steps, drawn by Nib. Read it here, or watch him draw it. Updated .
1phone guess
You already know a tiny version of this machine: your phone's autocomplete. Type 'See you' and it offers 'soon' or 'later'. It does not know your plans. It has simply seen which words tend to follow which. ChatGPT is the same basic idea, scaled up enormously.
Autocomplete is a small version of the same idea.
When your phone suggests 'soon' after 'See you', does it know when you'll meet?
Show the answer
No. It is only reflecting which word commonly comes next.
2scoring guesses
Give ChatGPT 'The capital of France is' and it doesn't look up an answer. It scores many candidate next pieces of text, and ' Paris' scores far above the rest. Each candidate gets a probability, a number saying how likely it is to come next, and the model picks from among the high scorers.
Each possible next piece gets a likelihood score.
a common mix-upIt looks facts up in a database like a search engine.
3the loop
Here is the key move. Once a piece is chosen, it is attached to the end of the text, and the whole longer text goes back in to predict the next piece. One guess becomes a word, then a sentence, then a whole essay, each piece built on everything before it.
Each guess is added to the text and fed back in.
After the model adds 'Paris', what does it feed back in to guess the next piece?
Show the answer
The original text plus 'Paris', the whole longer text.
4branching
Because the choice is made from likelihoods, not looked up, there is usually some variety. Ask the same question twice and the wording can differ, since a slightly different high-scoring piece may be picked early and every later guess builds on it. That is why answers feel fluent and flexible rather than canned.
Picking among likely pieces makes answers vary.
a common mix-upThe same question must always produce the identical answer.
5guess, add, repeat
So the whole chatbot is this one small act, guess the next piece, repeated thousands of times. The rest of this book opens up the three questions that act raises: what the 'pieces' are, how the guessing got so good, and why good guessing sometimes goes wrong.
One simple act, repeated, produces the whole conversation.
Every fluent reply you've ever read from a chatbot was built one guess at a time, so keep asking what makes each guess good, and when it isn't.