Backpropagation: how networks learn

Gradient descent, from Machine Learning Foundations, needs one thing: the gradient of the loss with respect to every parameter. For two parameters we could work that out by hand. A modern network has millions or billions, all tangled together through layer after layer. Backpropagation is the algorithm that computes all of those gradients efficiently — and it is nothing more than the chain rule from calculus, applied systematically.

Networks are computation graphs

Any network, however large, is built from simple operations — multiply, add, apply an activation, compute a loss. Draw each operation as a node and each value as an edge, and you get a computation graph. We will use the smallest interesting one: a single neuron with a sigmoid, trained on one example.

• Forward: $z = w \cdot x + b$, then $a = \sigma(z)$, then the loss $L = (a - y)^2$.
• We want $\partial L/\partial w$ and $\partial L/\partial b$: how much the loss changes when we nudge each parameter.

The chain rule, one node at a time

The chain rule says that if a change in $w$ affects $z$, which affects $a$, which affects $L$, then the total effect is the product of the individual effects:

Each factor is a local slope that only depends on one operation. Backpropagation computes them from the loss backwards, passing the running product from node to node.

Notice two things. First, the backward pass reused values from the forward pass — the sigmoid output $a$, the input $x$. Second, the gradient for $b$ came almost for free once we had $\partial L/\partial z$. In a big network, thousands of weights share the same intermediate gradients, which is why backpropagation computes all of them for roughly the cost of one extra forward pass.

Try it yourself

The calculator below runs the same neuron. Change the input, target and parameters, then take gradient steps and watch the loss fall. The numerical check at the bottom nudges $w$ by a tiny amount and measures the change in the loss directly — it always agrees with backpropagation.

Vanishing and exploding gradients

In a deep network, the gradient reaching the first layer is a product of many factors — one slope and one weight matrix per layer. Products of many numbers behave dramatically: if the factors are a little below 1, the product shrinks towards zero; a little above 1, and it grows without bound.

• With sigmoid activations, each layer multiplies the gradient by at most 0.25. Add layers and watch the early bars collapse — the first layers stop learning.
• With ReLU and standard (He) initialization, the bars stay level even at 40 layers.
• Make the initial weights too large and the gradients explode instead; too small and they vanish, even with ReLU.

Vanishing gradients

Gradients shrink as they travel back through many layers, so early layers learn extremely slowly.

Exploding gradients

Gradients grow layer by layer, causing huge, unstable updates and loss values of NaN.

Weight initialization

How weights are set before training. Schemes such as Xavier and He scale random weights by the layer width so signals keep a stable size.

Gradient clipping

Capping the size of the gradient before each update — a standard safeguard against explosions, especially in recurrent networks and transformers.

Residual connection

A shortcut that adds a layer’s input to its output, giving gradients a direct path backwards. It is why networks with hundreds of layers can be trained.

What the framework does for you

In practice you never write backpropagation by hand. Frameworks record the computation graph as your code runs and apply the chain rule automatically — this is called automatic differentiation:

import torch

w = torch.tensor(0.5, requires_grad=True)
b = torch.tensor(-0.3, requires_grad=True)
x, y = torch.tensor(2.0), torch.tensor(1.0)

a = torch.sigmoid(w * x + b)   # forward pass (the graph is recorded)
loss = (a - y) ** 2
loss.backward()                # backward pass: fills w.grad and b.grad

print(w.grad, b.grad)          # tensor(-0.2943) tensor(-0.1471)

Backpropagation tells us which direction to move each weight. How far to move — and how to avoid zig-zagging through narrow valleys — is the job of the optimizer, the subject of the next lecture.