1. Courses
  2. Mathematics for Machine Learning
  3. Lecture 4
Lecture 4 of 1418 min3D modelFree lecture

Derivatives and gradients

Training a model means walking downhill on a loss surface. Learn what a derivative measures, how the gradient generalizes it to many parameters, why gradient descent works — and why the learning rate matters — and how the chain rule makes backpropagation possible.

In this lecture you will

  • Interpret a derivative as the slope of a function and approximate it numerically
  • Compute partial derivatives and assemble them into a gradient
  • Explain gradient descent and the effect of the learning rate
  • Apply the chain rule and relate it to backpropagation

Every model in this course is trained the same way: define a loss that measures how wrong the model is, then adjust its parameters to make the loss smaller. The question is which way to adjust them. Derivatives answer that question: they measure how an output changes when an input changes a little. This lecture builds up from slopes to gradients to the chain rule — everything behind backpropagation.

The derivative is a slope

Take two points on a curve and draw the line through them: its slope is the average rate of change. Now slide the second point closer. As the gap h shrinks, the secant turns into the tangent, and its slope becomes the derivative:

f′(x)=lim⁡h→0f(x+h)−f(x)hf'(x) = \lim_{h \to 0} \frac{f(x + h) - f(x)}{h}
InteractiveFrom secant to tangentDrag the point; shrink h
Loading visual…

With h = 1 the secant (amber) is a poor estimate; at h = 0.001 it matches the tangent (teal). The lower panel plots f′: wherever f has a peak or a valley, f′ crosses zero.

A derivative of zero marks a flat spot — a peak, a valley or a plateau. Training looks for valleys of the loss, and the sign of the derivative says which way is downhill.

Derivative
The instantaneous rate of change of a function: the slope of its tangent line.
Partial derivative
The derivative with respect to one input while the others are held fixed.
Gradient
The vector of all partial derivatives. It points in the direction of steepest increase.
Learning rate
The step size of gradient descent: how far to move against the gradient each step.
Chain rule
The rule for differentiating a composition: multiply the derivatives of the parts.

Gradients: slopes in many directions

A real model has millions of parameters, so its loss is a surface over a huge space. With two parameters we can still see it. For each parameter, the partial derivative gives the slope in that direction; together they form the gradient ∇L, an arrow pointing uphill. Gradient descent takes a step the other way:

w←w−η ∇L(w)\mathbf{w} \leftarrow \mathbf{w} - \eta \, \nabla L(\mathbf{w})

where η (eta) is the learning rate.

3D modelGradient descent on a loss surfaceDrag to rotate
Loading 3D model…
  1. A loss surfaceThe height is the loss for each pair of parameter values. The contour rings on the floor join points of equal loss; the minimum is at the centre.
  2. The gradient fieldAt every point the gradient points uphill, perpendicular to the contours. It is longest where the surface is steepest.
  3. Step downhillFrom the starting point, the negative gradient (teal) is the direction of steepest descent.
  4. A good learning rateRepeating small steps traces a smooth path that curves into the valley and settles at the minimum.
  5. Too largeWith a learning rate near the stability limit, each step overshoots the steep direction, and the path zig-zags back and forth across the valley.

In practice, frameworks compute gradients automatically, and optimizers such as momentum and Adam adapt the steps to avoid zig-zagging — the Deep Learning course compares them side by side.

import numpy as np

def loss(w):  return 0.3 * w[0]**2 + 0.3 * w[0] * w[1] + w[1]**2
def grad(w):  return np.array([0.6 * w[0] + 0.3 * w[1], 2 * w[1] + 0.3 * w[0]])

w, lr = np.array([-2.7, 1.55]), 0.3
for step in range(25):
    w = w - lr * grad(w)           # step against the gradient
print(w, loss(w))                   # close to the minimum at (0, 0)

The chain rule

Models are chains of functions: a linear layer, then an activation, then another layer, then the loss. To know how the loss changes with a weight deep inside, we need the derivative of a composition. The chain rule says: multiply the rates along the chain.

dydx=dydu⋅dudx\frac{dy}{dx} = \frac{dy}{du} \cdot \frac{du}{dx}
AnimationThe chain rule, with gears
Loading animation…
  1. A chain of quantitiesGear x drives gear u, which shares an axle with a larger gear that drives y. Each quantity depends on the one before it.
  2. Rates multiplyWhen x turns once, u turns 3 times; when u turns once, y turns 2 times. So y turns 6 times for each turn of x.
  3. The ruleThat is the chain rule: dy/dx = dy/du · du/dx = 2 × 3 = 6.
  4. Local ratesFor curved functions the rates depend on where you are. At x = 1, with u = x² and y = sin u, the local rates are 2 and cos 1, so dy/dx ≈ 1.08.
  5. BackpropagationA network computes values forward, then multiplies local derivatives backwards from the loss to every weight. That backward pass is the chain rule, applied layer by layer.

Loss surfaces and gradients handle how models learn. The last lecture of the module covers how models deal with uncertainty: probability and Bayes’ theorem.

Key takeaways

  • The derivative f′(x) is the slope of f at x — the limit of the slope of a secant line as the step shrinks. At a peak or a valley it is zero.
  • With many parameters, the partial derivatives form the gradient, a vector that points in the direction of steepest increase.
  • Gradient descent repeatedly steps against the gradient; the learning rate sets the step size — too small is slow, too large overshoots or diverges.
  • The chain rule multiplies local rates of change along a chain of functions; backpropagation applies it layer by layer, backwards.

Check your understanding

4 questions · answers explained

Q1What does f′(3) = −2 tell you?

The derivative is the local slope. A slope of −2 means the function goes down roughly twice as fast as x goes up, near that point.

Q2For a loss L(w₁, w₂), what is the gradient?

Each partial derivative measures how the loss changes when one parameter moves and the others stay fixed. Together they form a vector that points uphill.

Q3Gradient descent zig-zags across a narrow valley and barely makes progress. What is the usual first fix?

Zig-zagging means the steps are too large for the steep direction. A smaller learning rate, momentum or an adaptive optimizer such as Adam smooths the path.

Q4If u = 3x and y = u², what is dy/dx at x = 1?

By the chain rule, dy/dx = dy/du · du/dx = 2u · 3. At x = 1, u = 3, so dy/dx = 6 × 3 = 18.