Rosehip Mole asked, and Nib — askNib's tutor — drew the answer live at a whiteboard. This is the spoken transcript; enable JavaScript to watch it drawn.
trust region policy optimization animated
Imagine tuning a robot's policy by gradient ascent. A big step might look great on paper but wreck the robot in reality. Why?
The problem: our estimate of how good a new policy is only stays accurate near the old one. Far away, the estimate lies to us.
So TRPO draws an invisible fence — a trust region — around the current policy, sized by how much the new policy is allowed to differ.
Inside that fence, TRPO finds the best possible step — as far as the constraint allows, but never one inch further.
Next round, the fence recenters on the new policy and shrinks or grows as needed — safe, steady climbing instead of a wild leap.
TRPO's trick is simple but powerful: trust your gradient, but only as far as your map of the world stays true.
How does PPO simplify TRPO's constraint?
TRPO's fence needs a hard KL constraint solved with second-order math — costly and fiddly. PPO asks: can we get the same safety cheaply?
PPO tracks the ratio of new policy probability to old policy probability for the action taken — call it r. At the old policy, r equals one.
Instead of a fence in policy-space, PPO just clips r directly between 0.8 and 1.2 — say within twenty percent — and takes the smaller, safer objective.
If the update tries to push r outside that band, the gradient signal flattens to zero — no reward for straying too far, no complex math needed.
PPO keeps TRPO's caution but trades the fancy fence for a simple clip — nearly as safe, far easier to compute.