Why the gradient, and why negative
Zoom in far enough on any smooth function and it looks like a plane. The gradient is that plane's slope — a vector whose direction is steepest ascent and whose length is how steep. Reverse it and you have the best local guess at "downhill".
The derivation, briefly
To decrease f as much as possible with a small step d, make the dot product as negative as possible. A dot product is minimized when the two vectors point in exactly opposite directions — so d ∝ −∇f.
What each piece contributes
- ∇f(θ)
- Direction and magnitude of steepest increase. Computed by backpropagation in a network, by hand or autodiff elsewhere.
- −
- Turn around. This is the entire distinction between gradient descent and gradient ascent.
- α
- Learning rate — how far to trust the local approximation. The single most important hyperparameter in machine learning.
- ←
- Overwrite and repeat. Nothing from the previous step is retained by plain gradient descent, which is exactly what momentum changes.
Three lines, and the decisions around them
# the entire algorithm theta = initialize() for step in range(max_steps): g = grad(loss, theta) # the only expensive part theta = theta - lr * g if norm(g) < tol: break
Initialization
Zeros work for linear and logistic regression and fail completely for neural networks — every neuron in a layer would compute the same thing forever. Networks need random initialization scaled to the layer width: He for ReLU, Xavier for tanh.
Stopping
Small gradient norm, small change in loss, a step budget, or — in practice, almost always — validation performance ceasing to improve. Training loss alone is the wrong stopping signal for anything you intend to deploy.
Cost per step
One gradient evaluation. Backpropagation computes it for roughly twice the cost of a forward pass regardless of parameter count, which is the fact that makes training large networks possible at all.
The one number that decides everything
Too small and you'll still be training next week. Too large and the step overshoots the valley, lands higher than it started, and each iteration makes things worse. The gap between those is often less than a factor of two.
Reading a loss curve
- slow decay
- Learning rate too small. Increase by 3× and try again.
- fast then flat
- Roughly right, possibly needs decay to squeeze out the last of it.
- bouncing
- Too large. It's making progress but overshooting each valley.
- rising / NaN
- Far too large. Divergence. Cut by 10× immediately.
- flat from step 0
- Not a learning rate problem. Check the gradient is reaching the parameters at all.
The theory, for a quadratic
With Hessian eigenvalues between λmin and λmax, gradient descent converges if and only if:
The largest curvature sets the speed limit, but the smallest sets how fast you actually make progress along the shallow direction. When those two differ by a factor of a hundred, you are forced to crawl along the direction that matters in order to stay stable along the one that doesn't. That is the ravine, stated in eigenvalues.
Batch, stochastic, mini-batch
The gradient of the loss over a dataset is the average of the per-example gradients. You are free to estimate that average from a subset — and doing so turns out to be better, not merely cheaper.
| Variant | Examples per step | Character |
|---|---|---|
| Batch GD | All of them | Exact gradient, smooth monotone descent, one update per epoch. Unusable past moderate data sizes |
| Stochastic GD | One | Very noisy, very fast per step, needs a decaying learning rate to settle |
| Mini-batch | 32–512, typically | The universal default. Enough averaging to be stable, small enough to be fast, and sized to fill the GPU |
Vocabulary that trips people up: an iteration is one parameter update, an epoch is one full pass through the training data. With 50,000 examples and a batch size of 100, one epoch is 500 iterations. Papers report both, and they are not interchangeable.
Why it's harder than the picture suggests
Ill-conditioning
The Hessian's condition number κ = λ_max / λ_min measures how elongated the bowl is. Plain gradient descent contracts the error by roughly (κ−1)/(κ+1) per step, so a condition number of 1,000 means you make about a tenth of a percent of progress per iteration. This is the dominant practical obstacle, and the hero's ravine is a mild version.
Saddle points
Points where the gradient vanishes but which are minima in some directions and maxima in others. In high dimensions they vastly outnumber local minima — a critical point needs every eigenvalue positive to be a minimum, and that gets exponentially unlikely as dimensions grow. Gradients near a saddle are tiny, so progress crawls.
Plateaus and cliffs
Large flat regions produce almost no gradient and stall progress. Sharp cliffs produce enormous ones and throw the parameters into nonsense in a single step. Recurrent networks are especially prone to both, which is why gradient clipping originated there.
Local minima
The classic worry, and mostly not the real problem. In large overparameterized networks most local minima found in practice have similar loss values, so which one you land in matters far less than the textbook picture implies. Saddle points and conditioning cost you much more.
Momentum, adaptivity, and Adam
Every improvement over plain gradient descent does one of two things: remember previous gradients, or give each parameter its own step size. Watch three optimizers run the same ravine from the same start.
| Method | Idea | Notes |
|---|---|---|
| Momentum | Accumulate a velocity: v ← βv − αg; θ ← θ + v | Oscillations cancel across steps, consistent directions build up. β = 0.9 is near-universal |
| Nesterov | Evaluate the gradient at where momentum is about to take you | A look-ahead correction. Slightly better, rarely decisive |
| AdaGrad | Divide by the accumulated sum of squared gradients | Per-parameter rates. Excellent for sparse features, but the rate decays to zero and stops |
| RMSProp | Same, with an exponential moving average instead | Fixes AdaGrad's terminal decay. Long the default for RNNs |
| Adam | Momentum and RMSProp, with bias correction | The default for almost everything. β₁=0.9, β₂=0.999, ε=1e-8 |
| AdamW | Decouples weight decay from the adaptive scaling | Adam's L2 penalty interacts badly with the per-parameter rates. AdamW is the fix, and is what you should use |
Changing the step size as you go
A rate large enough to make early progress is too large to settle precisely at the end. Nearly every serious training run therefore varies it over time.
| Schedule | Shape | When |
|---|---|---|
| Constant | Flat | Fine with Adam on short runs. The honest baseline |
| Step decay | Drop by 10× at set epochs | Classic vision recipes. Simple and effective |
| Exponential | Smooth continuous decay | When you want no discontinuities |
| Cosine annealing | Smooth decay to near zero | The modern default for deep learning. Often with warm restarts |
| Warmup | Ramp up over the first few hundred steps | Essential for transformers and large batches. Adam's variance estimates are unreliable early on |
| One-cycle | Up then down, with inverse momentum | Fast convergence on a fixed budget |
| ReduceLROnPlateau | Cut when validation stops improving | Reactive rather than scheduled. Useful when run length is unknown |
The cheapest fix available
Feature scaling is usually presented as tidying-up. It isn't. It directly reshapes the loss surface, and it is the highest-return intervention available to anyone using a gradient method.
What's actually happening
If one feature ranges over thousands and another over fractions, the loss changes enormously along one parameter axis and barely along the other. The contours become a long thin ellipse — a ravine, manufactured entirely by your choice of units.
Standardizing makes the two axes comparable, the contours round out, and the condition number collapses. Same problem, same optimum, dramatically easier descent.
Inside a network, normalization layers do the same job repeatedly. Batch norm, layer norm and their relatives keep activations well-scaled at every depth, which is why they let you train deeper networks with larger learning rates. Their original explanation has been disputed, but the effect on the optimization landscape is not in question.
When the signal doesn't arrive
Vanishing gradients
Backpropagation multiplies derivatives layer by layer. If each is below one, the product shrinks geometrically and early layers receive almost nothing — they simply stop learning while later layers train normally.
Fixes: ReLU-family activations instead of saturating ones, residual connections that give gradients a direct path, careful initialization, normalization layers, and gated architectures for sequences.
Exploding gradients
The same product running the other way. One enormous step and the parameters land somewhere that produces NaN, from which nothing recovers.
Fixes: gradient clipping by global norm — rescale the whole gradient if its norm exceeds a threshold, preserving direction while capping magnitude. One line, and it's standard practice for RNNs and transformers.
# the standard PyTorch step, with the guards in place optimizer.zero_grad(set_to_none=True) # gradients accumulate by default! loss = criterion(model(x), y) loss.backward() torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) optimizer.step() scheduler.step()
Guarantees, and their absence
Convex problems: solid ground
- Any local minimum is global. There is nowhere bad to end up.
- For an L-smooth function, α ≤ 1/L guarantees the loss decreases every step.
- Convergence rate O(1/t), improving to O(1/t²) with Nesterov acceleration.
- Strongly convex problems converge geometrically, at a rate set by the condition number.
Linear regression, logistic regression and SVMs all live here, which is why they train reliably and reproducibly.
Non-convex problems: almost nothing
For a neural network, theory promises convergence to a point where the gradient is small. It does not promise that point is good, or that two runs will find comparable ones.
Defaults, and what to check when it fails
Starting points that usually work
- optimizer
- AdamW. Switch to SGD+momentum only if you're going to tune it properly.
- learning rate
- 3e-4 for Adam on a new problem. 0.1 for SGD+momentum on vision with a schedule.
- momentum
- 0.9. Almost never worth tuning.
- batch size
- The largest that fits, then adjust the learning rate to match.
- schedule
- Linear warmup then cosine decay.
- clipping
- Global norm 1.0 for anything recurrent or transformer-shaped.
Loss won't go down — in order
- Overfit a single batch. If the model can't drive the loss to near zero on ten examples, the bug is in the model or the data pipeline, not the optimizer.
- Check the learning rate across three orders of magnitude before changing anything else.
- Verify gradients are non-zero and reaching every layer. Print gradient norms per layer.
- Confirm inputs are scaled and contain no NaNs.
- Check labels line up with inputs. A shuffling bug looks exactly like an optimization failure.
- Confirm zero_grad is being called and the loss is what you think it is.
Using curvature, and why we mostly don't
Gradient descent knows the slope. Second-order methods also know how the slope is changing, which lets them step directly toward the bottom of a local quadratic approximation instead of feeling their way there.
| Method | Step | Cost and verdict |
|---|---|---|
| Newton | θ ← θ − H⁻¹g | Quadratic convergence near the optimum. O(p³) per step and needs the Hessian — impossible for large models |
| Quasi-Newton (BFGS) | Builds an approximate inverse Hessian from gradients | O(p²) memory. Fine for thousands of parameters, not millions |
| L-BFGS | Keeps only the last few update pairs | Excellent for smooth full-batch problems. Degrades badly with mini-batch noise, which rules it out for deep learning |
| Natural gradient | Uses the Fisher information as the metric | Principled and expensive. K-FAC and similar make it approximately practical |
| Adam | Per-parameter scaling from squared gradients | A crude diagonal curvature estimate. Nearly free, which is why it won |
Drills
| # | Do this | You'll know it when |
|---|---|---|
| 1 | Minimize f(x) = x² by hand for five steps at three learning rates | You can predict divergence from α and the curvature |
| 2 | Implement gradient descent for linear regression in NumPy; match the normal equation | Both give the same β |
| 3 | Plot loss vs iteration for α across four orders of magnitude | You recognize each failure shape instantly |
| 4 | Run the same problem with unscaled then standardized features | The iteration count difference is startling |
| 5 | Add momentum to your implementation and re-run the ravine | You can explain the cancellation argument without notes |
| 6 | Implement Adam from the paper's pseudocode, bias correction included | It matches the framework version step for step |
| 7 | Compare batch, mini-batch and single-example descent on one dataset | You've seen the noise-versus-speed trade directly |
| 8 | Write a learning rate finder and run it on a real model | You stop guessing the learning rate |
| 9 | Deliberately break a training run six ways and diagnose each from the loss curve alone | Debugging becomes reading rather than guessing |
| 10 | Overfit a single batch on every new model you build, before anything else | It becomes automatic |
The one-paragraph summary
Gradient descent minimizes a function by repeatedly stepping against its gradient, scaled by a learning rate — an algorithm with no map, no memory and no knowledge of where the minimum lies, only the local slope. The learning rate is the decisive hyperparameter: too small and progress crawls, too large and the iterate overshoots and diverges, with the stability limit set by the largest curvature. Estimating the gradient from mini-batches rather than the full dataset is both faster and better, because the resulting noise helps escape saddle points and biases the search toward flatter minima. The dominant practical obstacle is ill-conditioning — elongated loss surfaces where the locally steepest direction is nearly perpendicular to the direction of real progress — which momentum addresses by accumulating consistent directions and cancelling oscillating ones, and which feature scaling addresses by reshaping the surface at source. Adam combines momentum with per-parameter step sizes and is the reasonable default; AdamW is its corrected form. Almost everything else — warmup, cosine decay, gradient clipping, normalization layers, careful initialization — exists to keep that three-line loop stable on surfaces it has no right to succeed on.