Nib asked for
Attention: Weighing Earlier Words
Attention lets every token check how relevant each other token is, so a word like "it" can pick up its meaning from the right noun.
5 steps, drawn by Nib. Read it here, or watch him draw it. Updated .
1the sentence
Take the sentence: "The trophy didn't fit in the suitcase because it was too big." To predict what comes next, the model needs to know what "it" means. But after tokenizing, "it" is just one token turned into a number, and that number is the same in every sentence. On its own, it carries almost nothing.
A token's number alone doesn't say what it means in this sentence.
a common mix-upThe model looks up a fixed meaning for each word, like a dictionary.
Why can't the model rely on the token "it" alone to know what it refers to?
Show the answer
The same token appears in countless sentences, so its number can't say which thing it points to here.
2scoring
Attention fixes this by letting each token look back across the other tokens and ask, "which of you matter for understanding me?" Every token gets a relevance score for each other token it can see. Think of it as a spotlight whose brightness varies: bright on the helpful words, dim on the rest.
Each token scores how relevant every other token is to it.
Does "it" give every other token the same weight?
Show the answer
No. Scores differ: some tokens get a lot of weight and others almost none.
3relevance
In our sentence, the scores for "it" come out high on "trophy" and "suitcase" and low on words like "the" and "in". Then "big" matters too: a trophy that is too big fails to fit, while a suitcase that is too big would not. The model learned these patterns in training, from countless sentences where such clues pointed to the right noun.
Learned patterns from training decide which words get the high scores.
a common mix-upThe model just links "it" to the nearest noun.
If the sentence ended "because it was too small," what should change?
Show the answer
The clue word changes, so the weight shifts toward "suitcase," the thing that being too small would block.
4absorbing
Next, the token blends in information from the words it weighted heavily, in proportion to their scores. So the number for "it" is updated into something richer: "it, in the sense of the trophy." The token is now a version of itself carrying context. Its meaning still depends on the whole sentence, and often that comes mainly from an earlier noun, though not always.
Attention blends in information from the weighted words, so "it" now carries context.
5many layers
This happens for every token at once, and it is repeated many times in stacked layers, so understanding builds up gradually. Each layer's updated tokens are fed into the next, which can ask better questions. By the end, the last token has gathered what it needs to predict the next piece of text, as page one described.
Every token does this, layer after layer, until the model can predict the next token.
What is the point of all this attention in terms of the model's basic job?
Show the answer
It gives each token the context it needs so the model can predict the next token well.
So when a chatbot gets "it" right, it's not looking up a meaning: every word has weighed the others, and context decided.