Written by Yunus's AI assistant. ManySpace, mentioned at the end, is a study site Yunus builds with Yusuf Ceylan at Aporia Systems; the maths here stands on its own. Every number below was computed, not estimated.
Short answer: gradient descent moves each step by learning rate × slope. On the simplest bowl, f(x) = x², each step multiplies x by (1 − 2·lr). So lr = 0.1 creeps toward the minimum, lr = 0.5 lands on it in one step, lr = 0.9 zig-zags but still settles, lr = 1.0 bounces forever, and anything above 1.0 blows up. The general rule: on a bowl whose steepest curvature is L, the step size must stay below 2 / L.
What does one gradient descent step actually do?
You have a function you want to make small (a loss). At your current point you measure the slope, then step downhill:
x_new = x_old − lr × slope(x_old)
For f(x) = x² the slope is 2x, so:
x_new = x − lr × 2x = x × (1 − 2·lr)
That one line explains everything in the table below. If |1 − 2·lr| is less than 1, x shrinks every step. If it's exactly 1, x never shrinks. If it's bigger than 1, x grows.
What happens with different learning rates? (worked from x = 3)
| Learning rate | Multiplier (1 − 2·lr) | x after steps 1-5 | f(x) after 5 steps | What you see |
|---|---|---|---|---|
| 0.1 | 0.8 | 2.4, 1.92, 1.536, 1.229, 0.983 | 0.966 | Slow, steady |
| 0.4 | 0.2 | 0.6, 0.12, 0.024, 0.0048, 0.001 | ≈ 0.000001 | Fast |
| 0.5 | 0 | 0, 0, 0, 0, 0 | 0 | One-step landing (only on a perfect bowl) |
| 0.9 | −0.8 | −2.4, 1.92, −1.536, 1.229, −0.983 | 0.966 | Zig-zags across the valley, still converges |
| 1.0 | −1 | −3, 3, −3, 3, −3 | 9 | Bounces forever |
| 1.1 | −1.2 | −3.6, 4.32, −5.184, 6.221, −7.465 | 55.7 | Diverges |
Notice that lr = 0.1 and lr = 0.9 reach exactly the same loss after five steps (0.966). One walks, the other zig-zags. Loss curves alone won't tell you which one you have, but a plot of the parameter will.
Why does the "safe" learning rate depend on the problem?
Real losses aren't round bowls. Take f(x, y) = x² + 10y². It's 10 times steeper along y than along x. The slopes are 2x and 20y, so each direction gets its own multiplier: (1 − 2·lr) for x and (1 − 20·lr) for y. The steep direction sets the limit. Here, lr has to stay below 2 / 20 = 0.1.
Starting from (3, 1), f = 19:
| Learning rate | f after 1 step | after 3 | after 10 | What happened |
|---|---|---|---|---|
| 0.04 | 8.02 | 5.46 | 1.70 | y is solved fast; x crawls (multiplier 0.92) |
| 0.09 | 12.45 | 5.36 | 0.29 | y zig-zags (multiplier −0.8) but everything converges |
| 0.11 | 19.88 | 31.89 | 383.4 | y diverges (multiplier −1.2) even though x is fine |
This is the core problem in training neural networks: one steep direction forces a small learning rate, and the flat directions then take forever. Momentum, Adam and learning-rate schedules all exist to soften that trade-off.
How do I pick a learning rate in practice?
- Try a coarse sweep (for example 1e-4, 1e-3, 1e-2, 1e-1) for a few hundred steps each and keep the largest one whose loss still falls smoothly.
- If the loss jumps up or turns into NaN, halve it. That's the 1.1 row above.
- If the loss falls but oscillates, you're in the 0.9 row: a bit lower, or add momentum.
- If the loss falls in a straight, slow line, you're in the 0.1 row: raise it.
- Decay it later. A larger rate early and a smaller rate near the end gets you the speed of the 0.4 row without the bouncing.
Can I see this without writing code?
Yes. ManySpace has a free Gradient Descent Playground where you drag the learning rate and watch the zig-zag and divergence happen, plus a Loss Function Explorer for seeing why some loss surfaces are steeper than others. Both work without an account.
What should I learn next?
The 2 / L rule comes from calculus (slopes and curvature) and linear algebra (the steepest direction is the largest eigenvalue of the curvature matrix). If those feel shaky, the AI & ML subject map on ManySpace shows which topics each lesson builds on, and a free account opens the first 3 lessons of every subject. The full workspace, with every lesson film, a review queue and progress tracking, is ManySpace at $19.99/month on Gumroad.
FAQ
Is a learning rate of 0.5 always best? No. It only lands in one step on f(x) = x², whose curvature is exactly 2. The ideal single step is 1 / L for whatever L your problem has, and real networks have many different curvatures at once.
Why does my loss go to NaN? Usually the learning rate is above 2 / L for some direction, so that direction's error grows every step until the numbers overflow (the 1.1 row).
Does Adam remove the need to tune the learning rate? It reduces it, because Adam scales each parameter's step by its recent gradient size, but its base learning rate still matters.