How does gradient descent avoid local minima?

1the myth

Quick correction first: plain gradient descent does not avoid local minima. It rolls downhill and stops where the slope is zero, even when a deeper valley sits nearby.

2stuck

Start on the right slope and the ball settles into the near valley. The deeper valley on the left is never found.

3the rule

Each step is: new position equals old position minus learning rate times slope. Zero slope means zero step, and the rule cannot see any farther.

4many dimensions

So why does training work? In millions of dimensions, a trap must curve upward in every direction at once. Picture a coin flip per direction: all heads is rare.

5the saddle

Most flat spots are saddles instead: a valley one way, a hill the other. The gradient is tiny there, so progress crawls, but a way down exists.

6what helps

And real training adds help. Mini-batch noise jostles the ball, and momentum keeps it rolling. Both can carry it off a saddle or out of a shallow dip.

7the whole idea

So it does not dodge local minima; in big networks it just rarely meets bad ones. Evidence suggests the minima it finds tend to be nearly as good as the best.