Overfitting and generalization
A model that aces its training data can still fail on day one in production. See overfitting and underfitting happen as you change model complexity, and learn the techniques teams use to build models that generalize.
In this lecture you will
- Explain the difference between training performance and generalization
- Recognize underfitting and overfitting from training and validation error
- Apply remedies such as more data, simpler models, regularization and early stopping
- Avoid data leakage when splitting and preparing data
In the last three lectures we measured a model by its loss on the training data. That is useful during training, but it is not what we actually care about. A model earns its keep on new data: tomorrow’s transactions, next month’s customers. How well it performs there is called generalization.
Memorizing is not learning
Imagine a student who memorizes last year’s exam answers word for word. They will score perfectly on last year’s paper and badly on this year’s. Models can do the same thing: with enough flexibility, a model can fit every training example exactly — including its noise and quirks — without capturing the real pattern underneath.
That is overfitting. Its opposite, underfitting, is a model too simple to capture the pattern at all, like fitting a straight line to a curve.
Try it: turn up the complexity
The black points are training data; the hollow amber points are validation data drawn from the same pattern, which the model never sees. Increase the polynomial degree and watch both the curve and the two error lines on the right.
Data is simulated from the dashed curve plus random noise, so we know the true pattern.
At low degrees the curve can’t bend enough, and both errors are high — underfitting. Around the middle the curve follows the true pattern and validation error is lowest. At high degrees the curve twists to pass through individual training points: training error keeps falling, but validation error shoots up. Switch on L2 regularization and see how it tames the wild curves even at high degrees.
Bias and variance
The two failure modes have names worth knowing:
- Bias
- Error from overly simple assumptions. High-bias models underfit: they miss the pattern even on training data.
- Variance
- Error from sensitivity to the particular training sample. High-variance models overfit: small changes in the data produce very different models.
- Bias–variance trade-off
- Making a model more flexible lowers bias but raises variance. The best model sits in between — and more data shifts that sweet spot toward more flexible models.
You can see both directly. Below, twelve different training sets are drawn from the same pattern, and a model of the chosen complexity is fitted to each one.
At degree 1 the twelve lines agree with each other but all miss the curve: bias. At degree 10 they wiggle in different directions: variance. Keep degree 10 and raise the number of points — the variance shrinks, which is why more data lets you afford a more flexible model.
Remedies for overfitting
When the gap between training and validation error is large, teams reach for a familiar toolkit:
- More data. The single most reliable fix. Noise averages out, and the real pattern dominates.
- A simpler model. Fewer features, lower degree, shallower trees or smaller networks.
- Regularization. Add a penalty for large weights to the loss. L2 regularization (ridge) shrinks weights smoothly; L1 (lasso) can push some to exactly zero, effectively selecting features.
- Early stopping. Watch validation error during training and stop when it starts rising.
- Cross-validation. Rotate which slice of data serves as validation, so decisions don’t depend on one lucky split.
- Lock the test setBefore anything else, a test set is set aside. It will be used once, at the very end.
- Five foldsThe remaining data is split into five equal folds. The first model trains on folds 2 to 5 and is scored on fold 1.
- RotateFour more models are trained, each validated on a different fold. Every example is used for validation exactly once.
- AverageThe average of the five scores is a steadier estimate than any single split, and the spread shows how much a single split depends on luck.
- Final modelAfter choosing settings by cross-validation, retrain on all the non-test data and evaluate once on the locked test set.
For underfitting, go the other way: add informative features, use a more flexible model, or reduce regularization.
Keep the test set sacred
Every time you look at validation results and adjust something, you tune the model a little toward that particular data. Over many iterations, the validation score becomes optimistic. That’s why a separate test set is held back and used once, at the end, to report the number you will stand behind.
Generalization after launch
Even a well-validated model can degrade once deployed, because the world changes. Customer behavior shifts, fraudsters adapt, a new product line appears. This data drift is why production models need monitoring: track the distribution of inputs and, as soon as true outcomes arrive, the model’s real accuracy — and retrain when they drift.
This completes the first module. You now have the core loop of machine learning: define a model, measure its loss, minimize it with gradient descent, and check that it generalizes. The next module applies that loop to the models used most in industry — trees, ensembles and neural networks.
Key takeaways
- The goal is low error on new data, not on the training set.
- Underfitting means both errors are high; overfitting means training error is low but validation error is high.
- More data, simpler models, regularization and early stopping all fight overfitting.
- Keep the test set untouched, and make sure no information from the future or the answer leaks into your features.
Check your understanding
3 questions · answers explainedQ1Your model has 2% error on training data and 25% error on validation data. What is the most likely diagnosis?
Q2Which of these is NOT a way to reduce overfitting?
Q3A churn model scores brilliantly in testing, then performs poorly in production. You discover a feature called “days since account closed”. What happened?