Linear regression and the loss function

Let’s build the simplest model that does something useful: predicting a flat’s price from its size. It seems too simple to matter, but linear models are still used every day in forecasting, pricing and analytics — and every idea in this lecture carries straight over to neural networks.

A model with two parameters

Plot past sales on a chart — size along the bottom, price up the side — and the points drift upward in a rough line. A linear model draws that line:

\hat{y} = w \cdot x + b

$x$ is the input feature — the flat’s size.
$\hat{y}$ (“y-hat”) is the model’s predicted price.
$w$ , the weight, is the slope: how much the price rises per extra square metre.
$b$ , the bias, shifts the whole line up or down.

Training this model means choosing good values for $w$ and $b$ . But “good” needs a precise definition first.

Measuring error: residuals

For each flat we know the actual price $y$ . The residual is the gap between reality and the prediction, $y - \hat{y}$ . In the chart below, residuals are the amber lines between each point and the line. Short lines mean a good fit.

To judge the whole model, we need to combine all the residuals into one number: the loss. The most common choice for regression is the mean squared error:

\text{MSE} = \frac{1}{n} \sum_{i=1}^{n} \left(y_i - \hat{y}_i\right)^2

Square each residual, then average them. The lower the MSE, the better the line fits the training data.

Try it: fit the line yourself

Drag the sliders to move the line and watch the loss change. Switch to squared errors to see what MSE really measures: the total area of the squares. Then let the computer find the best fit.

InteractiveFit a line and minimize the squared errorDrag the sliders

Loading visual…

The square root of MSE, called RMSE, is in the same units as the price, so it reads as a typical error.

You probably found a decent line quickly, then struggled to improve it further — changing the slope forces you to adjust the intercept, and vice versa. That tug-of-war between parameters is exactly what an optimizer resolves systematically.

Why square the errors?

Squaring does three useful things at once:

It removes the sign. Points above and below the line both count as errors instead of cancelling out.
It punishes big mistakes. An error of 10 costs 100, while ten errors of 1 cost only 10 in total. Many small errors are preferred to one large one.
It is smooth. The loss changes gradually as the parameters change, with no sharp corners. That is what makes it easy to minimize with the technique in the next lecture.

The downside is sensitivity to outliers: a few extreme values can drag the line toward them. When that is a problem, mean absolute error (MAE), which averages $|y - \hat{y}|$ , is more robust.

It helps to picture the loss as a function of the parameters. Below, the line passes through the origin, so it has a single parameter — its slope — and every slope has a loss. Drag the handle along the curve.

InteractiveEvery slope has a lossDrag the handle; switch the loss; add an outlier

Loading visual…

Squared error gives a smooth bowl with one lowest point, which is what makes it easy to minimize. Add the outlier: the best slope under MSE jumps noticeably, while under MAE it barely moves.

One feature is never enough

Real prices depend on more than size — location, floor, age, amenities. A linear model handles this by giving every feature its own weight:

\hat{y} = w_1 x_1 + w_2 x_2 + \dots + w_k x_k + b

Each weight tells you how much the prediction changes when its feature increases by one unit, holding the others fixed. That interpretability is why linear models remain popular in finance, insurance and healthcare, where decisions must be explained.

With two features, the line becomes a plane. Rotate the model to see it.

3D modelFrom a line to a planeDrag to rotate

Loading 3D model…

One featureSeen from the front, only the size of each flat is visible, and the best model is a line.
A second featureEach flat also has an age. Turning the view spreads the points out in depth: flats of the same size differ in price partly because of age.
A planeWith two features, the linear model is a plane. It is fitted the same way — by minimizing the squared vertical distances.
ResidualsThe red lines are the residuals: the vertical gaps between each flat’s real price and the plane. The loss adds up their squares.
Weights are slopesThe plane rises along the size axis and falls along the age axis. Each weight is the slope in one direction while the other feature is held fixed.

In this lecture you will