Gradient descent: learn the update, then test its limits

Gradient descent adjusts a model's parameters using the slope of its error. Measure the slope, take a small step downhill, and repeat. A good learning rate brings the error down; an oversized step can send it higher.

For a robot, the parameter might be a sensor calibration factor. A loss function measures how far its predictions fall from known distances. Gradient descent offers a way to improve that factor through repeated updates.

We will start with one parameter and a loss you can calculate by hand: f(x) = x². Its minimum sits at x = 0, where the loss is zero.

Turn the slope into an update

The derivative f′(x) measures a function's local slope. For x², the derivative is 2x. At x = 2, the slope is 4: a small move right increases the loss, so move left.

Gradient descent multiplies that slope by a positive learning rate, α, then subtracts the result:

The subscript t counts updates. The learning rate scales each update; it does not specify a fixed distance. With the same α, a large derivative produces a larger step.

With several parameters, collect their partial derivatives into a vector called the gradient, ∇f. The update becomes θ ← θ − α∇f(θ). Here, θ represents the whole parameter vector.

For a small change d, the loss changes by approximately ∇f · d. Choosing d opposite the gradient makes that dot product negative. This describes the steepest local decrease for a fixed Euclidean step length; a large step can leave the region where the approximation works.

Work through five steps

Start at x = 2 with α = 0.25. The first derivative is 4, so the update gives 2 − 0.25 × 4 = 1.

Recalculate the derivative at every new x:

Update	x before	Derivative 2x	x after	New loss x²
1	2	4	1	1
2	1	2	0.5	0.25
3	0.5	1	0.25	0.0625
4	0.25	0.5	0.125	0.015625
5	0.125	0.25	0.0625	0.00390625

Each step halves x. The loss falls to one quarter of its previous value because the loss squares x. After five updates, the loss has fallen from 4 to 0.00390625.

With exact arithmetic, these steps approach zero without reaching it in a finite number of updates. A practical implementation needs a stopping rule.

Test the learning rate

Choose “Steady steps” below and press Step five times. Check each update against the table. Run advances automatically until you pause it or the experiment reaches a stopping condition.

Interactive experiment

One rule, three loss curves

Choose a starting point and learning rate. Every update applies x ← x − αf′(x).

Loss functionBowl: x²Steeper bowl: 10x²Two valleys: (x² − 1)²

Starting x

2.0

Learning rate α

0.250

Iteration

0

Current x

2

Loss f(x)

4

Derivative f′(x)

4

Iteration 0: x = 2, loss = 4. Predict the next x, then press Step to check it.

StepRunReset

Reduced motion is on. Use Step to advance at your own pace.

Steady stepsOvershootToo largeFlat at top

Changing a control starts a new trace. The experiment stops at a small derivative, numeric limits, or 120 updates.

For this particular loss, substitute f′(x) = 2x into the update:

For any nonzero starting value, x approaches zero when |1 − 2α| < 1. Solving that inequality gives 0 < α < 1.

• α = 0.25: multiply x by 0.5 at each step.
• α = 0.5: reach zero in one step.
• α = 0.8: multiply by −0.6. Cross the minimum each time while getting closer.
• α = 1: alternate between x and −x forever, with unchanged loss.
• α = 1.1: multiply by −1.2. Each step moves farther from zero.

Try “Overshoot,” then “Too large.” Crossing the minimum can still reduce loss. Growing oscillations reveal the failure here. These limits belong to x²; other losses have different limits. Dive into Deep Learning explores this learning-rate behavior.

A flat slope can stop you at the top

Choose “Flat at top.” This sets the loss to f(x) = (x² − 1)² and starts at x = 0. Its derivative is 4x(x² − 1), which equals zero at that starting point.

Every plain gradient descent update would return the same x. Yet the loss there is 1, while x = −1 and x = 1 both give zero loss. The starting point is a local maximum.

Move the starting point to 0.2 and keep α = 0.1. Subtracting the derivative now moves x toward lower loss, and the updates approach the right valley. A start at −0.2 approaches the left valley.

The experiment stops when the derivative becomes very small. It also caps the number of updates and stops before extreme values break the numeric calculation. These guards bound the experiment; they do not certify a global minimum.

Scaling changes the safe step size

Switch to 10x². Its minimum is still at zero, but its derivative is 20x. With α = 0.25 and x = 2, the next x is 2 − 0.25 × 40 = −8. The rate that worked for x² now sends the loss upward.

For this steeper bowl, the update multiplier is 1 − 20α. Convergence from a nonzero start requires 0 < α < 0.1. Setting α = 0.025 reproduces the earlier sequence of x values.

Real models can have steep directions and shallow directions at once. Different feature scales can contribute to that imbalance. Standardizing suitable numeric features can help optimization, though it does not solve every conditioning problem.

Estimate scaling values from the training set, then apply those same values to validation and test data. Scikit-learn's preprocessing guide explains standardization.

Connect the update to training data

The linear regression lesson builds a loss from sensor calibration readings. Each prediction depends on a slope and an intercept. Changing either parameter changes the residuals, which changes the loss you want to reduce.

In supervised learning, the loss usually combines errors across examples. The amount of data used for each gradient gives three common variants:

• Batch gradient descent computes each update from the full training set.
• Stochastic gradient descent uses one sampled example per update.
• Mini-batch gradient descent uses a subset of examples per update.

Smaller batches can make updates cheaper and noisier. A noisy training-loss curve needs different interpretation from the exact, smooth losses in this experiment. Google's hyperparameter lesson compares these batch choices.

Methods such as momentum and Adam modify the update using information from earlier gradients. Their usefulness depends on the problem and settings. Start by understanding the plain update before comparing optimizers.

Run the calculation in Python

This example uses only Python's built-in features. Each line reports the updated x and its new loss.

x = 2.0
learning_rate = 0.25

for step in range(1, 6):
    gradient = 2 * x
    x = x - learning_rate * gradient
    print(f"{step}: x={x:.5f}, loss={x*x:.8f}")

Expected output:

1: x=1.00000, loss=1.00000000
2: x=0.50000, loss=0.25000000
3: x=0.25000, loss=0.06250000
4: x=0.12500, loss=0.01562500
5: x=0.06250, loss=0.00390625

Try it yourself

Exercise 1. For f(x) = x², start at x = 2 with α = 0.8. Calculate the next two x values and their losses. Does crossing zero imply failure?

Show solution: overshoot with progress

The first update gives 2 − 0.8 × 4 = −1.2, with loss 1.44. The next gives −1.2 − 0.8 × (−2.4) = 0.72, with loss 0.5184.

Both steps cross zero. Both reduce the loss from its previous value. The multiplier −0.6 changes the sign while shrinking the distance to zero.

Exercise 2. For f(x) = 10x², which learning rate reaches zero from x = 2 in one update? Explain why the same rate behaves differently on x².

Show solution: account for the steeper derivative

Solve 2 − α × 40 = 0 to get α = 0.05. Equivalently, the update multiplier 1 − 20α becomes zero.

For x², the derivative at 2 is only 4. The same learning rate gives 2 − 0.05 × 4 = 1.8. The loss's scale changes the step even when the learning rate stays fixed.

Use the conditional probability lesson next to separate a model's score from the evidence it gives you about a real event.

Sources and further study

• Dive into Deep Learning: Gradient descent: derivatives, learning rates, and local behavior.
• Google Machine Learning Crash Course: Hyperparameters: learning rates and batch size.
• Scikit-learn: Preprocessing data: standardization and feature scaling.