Nib asked, and Nib — askNib's tutor — drew the answer live at a whiteboard. This is the spoken transcript; enable JavaScript to watch it drawn.
Why can't AI count the r's in strawberry? (for a curious adult who uses chatbots but has never studied computing). Cover, in this order: What does an AI actually receive when you type a word?; Where do tokens come from?; Why is strawberry split into pieces the model can't see inside?; How does a model learn spelling at all?; Why does letting it write the letters out fix the count?; What else do tokens explain? — page 4: How Spelling Gets Learned
A chunk is only an ID number. Nothing inside berry tells the model it is spelled b, e, r, r, y; that spelling has to come from elsewhere.
Training text sometimes spells words out, like berry is b-e-r-r-y. There the letters appear as separate tokens beside the chunk, and the model links them.
Other clues help too: typos split words differently, rhymes share endings, and capitals leave traces. They are useful hints, but not a reliable spelling table.
To count, the model must recall letters for each chunk and add them. If its memory of berry is fuzzy, the total comes out wrong.
Here is the whole picture: spelling facts reach a chunk only through faint, indirect traces, so counting by recall can slip.
So the model isn't being careless; it recalls spelling from faint, indirect traces, which is exactly why spelling the word out can help.