Calculus I · When You Can’t Solve It, Chase It

The two methods were the same idea all along

We learned Newton’s method to find where a function is zero, and gradient descent to find where a function is smallest. Now watch them collapse into one.

A minimum of $f$ is a place where $f'(x) = 0$. But “find where $f'$ is zero” is a root-finding problem — it is exactly the job Newton’s method does, applied to the function $f'$ instead of $f$. So take Newton’s formula and replace $f$ with $f'$ (which means replacing $f'$ with $f''$, the derivative of the derivative — the curvature from Chapter 7):

$$x_{n+1} = x_n - \frac{f'(x_n)}{f''(x_n)}.$$

Now line up the two ways to hunt for a minimum:

$$\text{gradient descent:}\quad x_{n+1} = x_n - \eta\,f'(x_n) \qquad\qquad \text{Newton's:}\quad x_{n+1} = x_n - \frac{1}{f''(x_n)}\,f'(x_n).$$

They are the same shape. Both step against the slope $f'(x_n)$. The only difference is the step size in front. Gradient descent uses a fixed number $\eta$ that you pick and hope is good. Newton’s method uses $1/f''(x_n)$ — it reads the curvature off the function and sets its own step size automatically, taking small steps where the valley is sharply curved and big steps where it is gently curved. That is why Newton is so fast: it is not guessing the step size, it is computing the right one.

This even explains the magic $\eta = \tfrac12$ from a moment ago. For $f(x) = x^2$ the curvature is $f'' = 2$, so Newton’s automatic step size is $1/f'' = 1/2$ — exactly the $\eta$ that jumped to the answer in one step. Gradient descent with the perfect hand-tuned rate is Newton’s method, for that function.

So why does AI use the slow one?

Here is the honest payoff, and it is one of the central facts about how modern AI is actually built. If Newton’s method is so much faster — picks its own step size, converges in a handful of iterations — why does essentially every neural network on the planet get trained with plain gradient descent and its fixed, fiddly learning rate?

Because of what $f''$ becomes when $x$ is not a single number but a list of millions or billions of numbers — which is exactly the situation for a neural network, whose “position” is the gigantic list of all its adjustable parameters. With one variable, $f''$ is one number. With $N$ variables, the “curvature” is no longer a number; it becomes a square table with $N$ rows and $N$ columns — $N^2$ entries. For a model with a billion parameters, that table has a billion squared — a billion billion — entries. You cannot store it. You certainly cannot do the division (which becomes something far more expensive than ordinary division). Newton’s method, the fast one, becomes physically impossible at the scale that matters.

Gradient descent only ever needs the slope $f'$ — which, for $N$ parameters, is just $N$ numbers, one downhill direction per parameter. That you can compute, even for a billion parameters. So the field made a trade it did not exactly choose: give up Newton’s speed and self-tuning, accept a learning rate you have to babysit, in exchange for a method that is cheap enough to actually run. That trade — fast-but-impossible second-order versus slow-but-affordable first-order — is why “tuning the learning rate” is a phrase every person who trains models knows in their bones. You now know why it exists, from a parabola you analyzed with Chapter 1’s geometric sequences.

All 12 chapters of Calculus I · Foundations · All courses