Regularization and generalization

Deep networks have enormous capacity. A network with a few thousand parameters can draw almost any boundary you like — including one that snakes around every noisy, mislabeled training example. That is overfitting: excellent training scores, disappointing results on new data. This lecture is about the tools that keep networks honest.

See it happen

The playground below trains a real neural network in your browser. Filled dots are training data; hollow rings are test data the network never trains on. The background shows the network’s decision: teal where it predicts one class, amber for the other.

Try these experiments in order:

1. Spiral, 2 layers of 8, tanh. Press Train and watch the boundary wind around the arms. Both losses fall together — the model is learning the real pattern.
2. Raise the label noise to 20–30% and use 3 layers of 16. Train for a while. The training loss keeps falling as the boundary grows islands around mislabeled points, while the test loss turns upwards: overfitting.
3. Now add L2 weight decay 0.003 or dropout 30%, press Reset and train again. The boundary stays smoother and the test loss stays lower.
4. Circle with 1 layer of 2 neurons. The network cannot bend enough; both losses stay high: underfitting. Add neurons until it fits.

Generalization

How well a model performs on new data drawn from the same source as the training data. The only performance that matters in production.

Overfitting

The model fits noise and quirks of the training set. Training loss is low; validation loss is high or rising.

Underfitting

The model is too simple or undertrained to capture the pattern. Both training and validation loss stay high.

Validation set

Data held out from training and used to choose models, hyperparameters and when to stop.

Test set

Data touched only once, at the very end, for an honest final estimate.

Weight decay: prefer small weights

L2 regularization, usually called weight decay, adds a penalty for large weights to the loss:

The gradient of the penalty is $2\lambda w$, so every update also shrinks each weight a little towards zero. Weights survive only if the data keeps pushing them up. Networks with small weights change their outputs gently as the input changes — they produce smoother boundaries that tend to generalize. The strength $\lambda$ is a hyperparameter: too little does nothing; too much causes underfitting.

Dropout: train an ensemble in one network

Dropout takes a different route. At each training step it switches off a random fraction of the hidden neurons — typically 10–50%.

Early stopping: quit while you are ahead

The simplest regularizer of all is to stop training at the right moment. Validation loss usually falls, reaches a minimum and then rises as the model starts to memorize. Early stopping keeps the checkpoint with the best validation loss.

Move the marker to see the cost of stopping too early or too late, then increase the regularization strength: the turn in the validation curve comes later and lower, and the gap between the curves shrinks.

Other tools in the box

• More data is the most reliable cure for overfitting. When collecting more is impossible, data augmentation creates new training examples by transforming existing ones — flipping and cropping images, or paraphrasing text. You will see this in the Computer Vision course.
• Batch normalization normalizes each layer’s inputs within a mini-batch. It mainly speeds up and stabilizes training, with a mild regularizing side effect.
• Smaller or simpler models reduce capacity directly — sometimes the right answer.
• Label smoothing trains on slightly softened targets (0.9 instead of 1) so the network does not become overconfident.

model = nn.Sequential(
    nn.Linear(784, 256), nn.ReLU(), nn.Dropout(p=0.3),
    nn.Linear(256, 128), nn.ReLU(), nn.Dropout(p=0.3),
    nn.Linear(128, 10),
)
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01)

best = float("inf")
for epoch in range(100):
    train_one_epoch(model, optimizer)          # model.train() inside
    val_loss = evaluate(model)                 # model.eval() inside
    if val_loss < best:
        best, bad_epochs = val_loss, 0
        torch.save(model.state_dict(), "best.pt")   # keep the best checkpoint
    else:
        bad_epochs += 1
        if bad_epochs >= 10:                    # patience
            break

You now have the complete toolkit of a working deep-learning engineer: neurons and layers, activations, backpropagation, optimizers and regularization. The rest of the course applies it to the architectures that power modern AI — starting with convolutional networks for images.