How does backpropagation work in neural networks?

1the guess

A tiny network takes x equals 2 and guesses 3, but the right answer is 1. The loss, the squared error, is 4. Two weights sit in the chain.

2the question

Which weight deserves the blame, and how much? Backpropagation computes that for every weight at once. It only finds the blame; it changes nothing itself.

3backward start

Now go backward, from the loss. The loss is the guess minus the target, squared, so its slope with respect to y is 2 times 3 minus 1, which is 4.

4chain rule

Chain rule: multiply slopes as you walk back. Going through w2, h gets 4 times 0.5, which is 2. And w2 itself gets 4 times h, which is 24.

5last weight

One more step back. The slope at h is 2, and h came from x times w1, so w1 gets 2 times x, which is 2 times 2, which is 4.

6the update

Now gradient descent finally does the learning: each weight moves against its slope. With a rate of 0.01, w1 becomes 2.96 and w2 becomes 0.26, and the loss drops from 4 to about 0.29.