Nib asked for
Understanding and Trust
Whether or not the model truly understands, its output is a fluent, useful draft that you should verify, especially for facts, numbers, quotes, and citations.
6 steps, drawn by Nib. Read it here, or watch him draw it. Updated .
1the big question
By now you know the machine's core move: it predicts likely next tokens, shaped by training on huge amounts of text and by human feedback. Yet its answers often look insightful. So people ask whether it understands. Careful researchers genuinely disagree, and nobody has settled it.
Whether it understands is an open question, not a settled one.
a common mix-upThat the question has a known answer, either 'it's just autocomplete' or 'it thinks like us'.
2both sides
Each side has a point. Skeptics note that the model learned only from text and was never trained to check claims against the world, so fluent output can be hollow. Others note that predicting text well may force a model to build useful internal representations of how things work, which looks like a partial kind of understanding. Both can be partly true.
Both camps have real evidence; neither has won.
3same either way
Here is the useful part: you don't need to settle the debate to use the tool well. Whether or not understanding is happening, the behavior you'll meet is the same. The model produces fluent, confident text that is usually helpful and sometimes wrong, and it sounds equally sure either way.
Your trust decision doesn't depend on the philosophy.
a common mix-upThat a smarter-sounding answer is a more reliable one.
4sounds the same
Recall page 5: the model is trained to produce plausible text, not to check facts. A made-up date, a statistic, a quotation, or a reference to a paper that doesn't exist reads exactly like a true one. Nothing in the wording signals the difference, because the same process generated both.
A false detail looks just like a true one.
If a chatbot cites a study with an author, year, and journal, does the polish of that citation show it is real?
Show the answer
No. Fabricated citations look just as polished as real ones, so you must look the study up.
5check these
So verify hardest where a wrong detail is both likely and costly: facts, numbers, quotes, and citations. Those are the specific details the model reconstructs from patterns instead of looking up, so errors hide there. For a quote, find the original source. For a number, find where it came from. For a citation, confirm the work exists and says what is claimed.
Check the specifics: facts, numbers, quotes, citations.
a common mix-upThat checking means distrusting everything equally; the risk is concentrated in specific details.
6draft then verify
Treated as a draft, the model is genuinely valuable: it can outline, rephrase, brainstorm, explain a concept, summarize text you paste in, and get you moving fast. Use it for speed and a starting point, then be the editor who confirms what matters. You supply the checking; it supplies the first version.
Use it as a fast first draft, then verify.
You ask for a summary of a long email you pasted in, and for a statistic to support your report. Which needs more checking, and why?
Show the answer
The statistic. The summary draws on text in front of the model, while a recalled statistic may be invented and unverifiable by its tone.
So hold both ideas: it may be doing something that deserves a respectful question mark, and it can still be confidently wrong. Use it as a brilliant, fast draftsperson, and keep your own hand on the facts.