Nib asked for
Why Plausible Can Be Wrong
The model is trained to produce text that sounds likely, not text that has been checked, so it can state false details just as fluently as true ones.
5 steps, drawn by Nib. Read it here, or watch him draw it. Updated .
1the scoreboard
Recall the training game from earlier: the model guesses the next token, then nudges its numbers toward whatever would have been right. Notice what the score measured. It rewarded text that matched real writing. It never asked whether the sentence was true, only whether it was the kind of thing a person would plausibly write next.
Training rewarded 'sounds like real text,' not 'is true.'
a common mix-upThe model was trained to be correct. It was trained to be plausible; correctness only came along when true text happened to be the likely text.
2built piece by piece
Now picture it answering a question about a book you've never heard of. It has no lookup step. It builds the answer one token at a time, each chosen because it fits what came before. A title, an author, a year: each slot gets filled with something that fits the pattern of such answers.
Answers are built token by token from patterns, not retrieved from a record.
3two citations
Ask for a source on an obscure claim, and the likeliest continuation looks like a citation: a plausible surname, a journal name that exists, a year in the right range. The model fills the shape of a real answer. Nothing in it knows the paper doesn't exist, so the invented one reads exactly as smoothly as a real one.
A made-up citation has the right shape, so it comes out looking real.
4same confidence
There is also no built-in alarm for 'I'm out of my depth.' The model always produces some next token, and fluent wording carries the same confident tone whether the facts underneath are solid or invented. That is why a hallucination rarely looks hesitant or broken.
Confidence in the wording tells you nothing about whether it's accurate.
a common mix-upIf it sounds sure, it must know. Fluency and certainty of tone are produced the same way whether or not the facts are right.
If a chatbot gives you a detailed, confident quote from a famous speech, does the confidence tell you the quote is accurate?
Show the answer
No. The confident tone comes from fluent pattern-matching, so a detailed quote can still be partly or wholly invented.
5the widening gap
Hallucination is most likely where the training text was thin: obscure people, exact numbers, quotes, citations, recent events. Common, well-covered facts appear in many contexts, so the likely answer and the true one usually coincide. When they diverge, the model follows likely.
Errors cluster on rare, specific details where likely and true come apart.
So a chatbot isn't lying or broken when it invents a detail; it's doing exactly what it was trained to do. Treat fluent as a style, never as proof.