Models learn by rolling downhill on a loss surface. Each step: compute the gradient (slope), move opposite to it. Learning rate controls step size — too small and you crawl, too big and you overshoot.
Position x—
Loss f(x)—
Gradient f'(x)—
0.20
Try crank η up to 1.5 on the convex surface and step — it will overshoot and bounce around. That's why deep learning uses small η + many steps. On non-convex landscapes (right), the start point determines which local minimum you fall into.