Gradient descent: finding the bottom of the valley

In the last lecture you adjusted a line by hand until its error was small. That works for two parameters. A modern language model has billions. We need an automatic way to find good parameters — and almost all of machine learning uses the same one.

The loss landscape

Take a model with two parameters, a weight $w$ and a bias $b$ . For every possible pair of values we can compute the loss on the training data. Plot that loss as a height above the $(w, b)$ plane and you get a surface: high where the model is bad, low where it is good.

Training is now a geographic problem: find the lowest point in the landscape. For linear regression with squared error, the surface is a smooth bowl with a single bottom. Neural networks produce far more complicated landscapes, but the method for walking downhill is the same.

Walking downhill with the gradient

Imagine standing on the surface in thick fog. You can’t see the valley, but you can feel the slope under your feet. The sensible move is to take a step in the direction that goes down most steeply — then feel the slope again and repeat.

The gradient, written $\nabla L$ , is exactly that slope information: a vector that points in the direction in which the loss increases fastest. So we step the other way:

\theta \leftarrow \theta - \eta \, \nabla L(\theta)

Here $\theta$ stands for all the parameters (for us, $w$ and $b$ ), and $\eta$ — the learning rate — sets how big each step is. Repeat this update many times and the parameters roll downhill toward a minimum.

Try it: release the ball

The surface below is a real loss function. The ball is the model’s current parameters, and each dot is one gradient-descent step. Release it with the default learning rate, then experiment:

3D modelGradient descent on a loss surfaceDrag to rotate · click the surface to move the start

Loading 3D model…

The bowl is steeper in one direction than the other, as real loss surfaces usually are. That is why large learning rates zig-zag across the valley.

At 0.02, progress is painfully slow — the ball is still descending when we stop after 60 steps.
Around 0.10–0.30, it glides smoothly into the valley.
Around 0.45, it overshoots the steep direction and zig-zags from side to side, wasting steps.
Above 0.50, each step overshoots by more than the last. The loss grows without bound: training has diverged.

Choosing the learning rate

The learning rate is the single most important setting in training. There is no universally correct value; it depends on the model, the data and how the features are scaled. In practice teams:

start from a value known to work for similar models and try a few values around it, a factor of about three apart;
plot the loss against training steps and read its shape — a flat line means too small, a jagged or exploding line means too large;
use learning-rate schedules that start larger and shrink over time, and adaptive optimizers such as Adam, which adjust the step size per parameter automatically.

Watch the individual steps on a single parameter. Each step moves by the learning rate times the slope, so steps shrink naturally as the curve flattens near the bottom.

InteractiveTwenty steps of gradient descentSet the learning rate, then run

Loading visual…

On this curve, a learning rate of 1.0 lands on the minimum in a single step. Between 1 and 2 the steps overshoot and bounce from side to side; above 2 every bounce is bigger than the last. The exact limits depend on the curvature of the loss.

From the full dataset to mini-batches

Computing the exact gradient means looking at every training example — fine for a thousand rows, hopeless for a billion. So real systems estimate it:

Batch gradient descent: Uses the whole dataset for every step. Accurate but slow and memory-hungry.
Stochastic gradient descent (SGD): Uses one random example per step. Very cheap and very noisy.
Mini-batch gradient descent: Uses a small random batch — typically 32 to a few thousand examples. The standard choice: efficient on GPUs, with manageable noise.
Epoch: One full pass through the training data. Training usually runs for many epochs.

Compare the three on the same problem: fitting a line to 200 points, with the same learning rate and the same budget of examples processed.

InteractiveFull batch, mini-batch and SGDChange the budget; show or hide methods

Loading visual…

With 1,600 examples processed, full-batch descent has taken only 8 careful steps and is still far from the best fit. SGD has taken 1,600 noisy ones and is close — it keeps jittering around the minimum. Mini-batches get most of the progress with far less noise.

The noise from mini-batches is not all bad. It jiggles the parameters around, which can help them escape shallow dips and flat regions in the complicated landscapes of deep networks.

Local minima and other hazards

Our bowl has one bottom, so gradient descent always finds it. Deep-learning landscapes have many dips, plateaus and saddle points. In practice, very large models turn out to have many solutions of similar quality, and techniques such as momentum and adaptive optimizers keep training moving through flat stretches.

Gradient descent works for any model whose loss we can differentiate — including the classifiers of the next lecture, which predict categories instead of numbers.

In this lecture you will