Nib asked, and Nib — askNib's tutor — drew the answer live at a whiteboard. This is the spoken transcript; enable JavaScript to watch it drawn.
How do AI language models like ChatGPT actually work? (for a curious adult who uses chatbots but has never studied computing). Cover, in this order: How does ChatGPT work?; How does it read your words?; How does it learn which words come next?; How does it know which earlier words matter?; Why do AI chatbots hallucinate?; Why does it forget what you said earlier?; How did a text predictor learn to follow instructions?; Does it understand, and how much should you trust it? — page 7: Teaching It to Follow Instructions
Before any tuning, the model only continues text. Ask it a question and it may simply write more questions, like a quiz, instead of answering.
So developers gather thousands of examples: a request, then a good answer. The model trains on these the same way, guessing next tokens.
Next, the model writes several answers to one prompt. People rank them, best to worst. Those rankings teach a scoring signal for what people prefer.
Next, the model's numbers are nudged toward replies that score well and away from ones that score badly. Round after round, it drifts toward helpful.
Underneath, nothing changed: it still guesses one token at a time. But now the most likely continuation of your request is a real attempt to help.
A raw predictor is first taught from examples, then steered by rankings of good and bad answers, and it still guesses one token at a time.
So the helpful chatbot you talk to is a text predictor, taught by examples and human preferences to aim at what you want, which is why it can be so useful and still be wrong.