Nib asked for
How Spelling Gets Learned
The model never sees inside a chunk, so it learns which letters a chunk contains only indirectly, from patterns in text, and those facts stay weakly known and easy to get wrong.
5 steps, drawn by Nib. Read it here, or watch him draw it. Updated .
1just a number
Page 3 left us with a puzzle: if 'straw' and 'berry' are each a single number, how could the model ever know what letters are inside them? The number itself carries no letters. It is just an ID, like a locker number that says nothing about what is in the locker.
A token's number says nothing about its letters.
a common mix-upThe model can look inside a token and read off its letters.
If a token is only a number, can the model read its letters straight off it?
Show the answer
No. The number is an ID with no letters inside; letter facts must be learned some other way.
2spelled-out text
So where does the knowledge come from? Mostly from text that spells words out: spelling lessons, acronyms, word games, dictionary-style entries, someone writing 'berry is spelled b-e-r-r-y.' In that text the letters appear as separate tokens, right next to the chunk, so the model can pick up an association between the two.
Most letter knowledge comes from text that spells words out.
Why would a rarely spelled-out word be harder for the model to spell?
Show the answer
It has seen few examples linking that chunk to its letters, so the association is thin.
3weak clues
There are other indirect clues too. The same word sometimes gets split differently, for instance when it is misspelled or rare, so the model glimpses its pieces in other combinations. Rhymes, shared endings, and capitalization patterns hint at letters as well. But these are hints, not a lookup table.
Other clues exist, but they are hints, not a lookup table.
a common mix-upSpelling is learned only from lessons that say the letters aloud, or else it is fully reliable; really it is mostly spelling-text plus weaker clues.
4the slip
The result is a patchy memory. For common words that are spelled out often, the model's recall is usually good. For others, the link between chunk and letters is faint, so asking 'how many r's in strawberry?' means recalling letter facts about 'straw' (one r) and 'berry' (two r's) and adding them. A slip anywhere gives a wrong total.
Weak letter facts, combined from chunks, can add up wrong.
a common mix-upA wrong count means the model can't do simple arithmetic; the weak link is its recall of letters in each chunk.
What is the correct r count for straw plus berry?
Show the answer
Three: one in 'straw' and two in 'berry'.
5the whole idea
Here is the whole picture: spelling facts reach a chunk only through faint, indirect traces, so counting by recall can slip.
So the model isn't being careless; it recalls spelling from faint, indirect traces, which is exactly why spelling the word out can help.