Classification and logistic regression

Regression predicts a number. Just as often, the business question is a choice between categories: Is this payment fraudulent? Will this customer cancel next month? Is this email spam? That is classification, and the classic starting point is a model with a misleading name: logistic regression.

From score to probability

Start with the same linear score as before, $z = w \cdot x + b$ . It can be any number, from very negative to very positive. We want a probability between 0 and 1 instead. The sigmoid function squashes any score into that range:

\sigma(z) = \frac{1}{1 + e^{-z}}

Large positive scores map close to 1, large negative scores close to 0, and a score of 0 maps to exactly 0.5. The weight controls how steeply the curve rises; the bias shifts where it crosses the middle.

AnimationFrom score to probability to decision

Loading animation…

Raw scoresA linear model scores eight customers. The scores can be any number, which makes them hard to interpret or compare.
The sigmoidThe S-shaped sigmoid curve takes any score and returns a number between 0 and 1.
ProbabilitiesReading each score up to the curve and across gives a probability: 1.3 becomes 0.79, −0.6 becomes 0.35.
A thresholdAt a threshold of 0.5, customers above the line get a “yes”. Raising the threshold to 0.8 leaves only the most confident cases.
Steeper means surerA larger weight makes the curve steeper: the same scores now give probabilities closer to 0 or 1. Training chooses the weight and bias that fit the labels best.

From probability to decision

A probability is not yet a decision. To act — block the payment, call the customer — we need a threshold: predict “yes” when the probability is at least, say, 0.5.

The point where the probability crosses the threshold is the decision boundary. With one feature it is a single point on the axis; with two features it is a line; with many, a flat surface through the feature space. Logistic regression always draws a straight boundary, which is both its limitation and a big part of why it is easy to explain.

Try it: pass or fail?

Each dot is a trainee: how many hours they practised, and whether they passed a certification exam (filled) or not (hollow). Shape the curve with the first two sliders, then move the threshold and watch the mistakes shift between the two kinds of errors.

InteractiveLogistic regression and the decision thresholdAdjust the curve and the threshold

Loading visual…

The data is simulated for illustration. Red outlines mark wrong predictions at the current threshold.

Notice that the fitted curve doesn’t change when you move the threshold — only the decisions do. Choosing a threshold is a business decision, not a modelling one.

Two kinds of mistakes

A classifier can be wrong in two different ways, and they rarely cost the same:

True positive (TP): Predicted yes, and it was yes — a fraud caught.
False positive (FP): Predicted yes, but it was no — a genuine customer blocked.
True negative (TN): Predicted no, and it was no — a normal payment allowed.
False negative (FN): Predicted no, but it was yes — a fraud missed.

From these four counts come the metrics you will see in every model review:

Accuracy = (TP + TN) / everything — the share of all predictions that were right.
Precision = TP / (TP + FP) — when the model says yes, how often is it right?
Recall = TP / (TP + FN) — of all the real yeses, how many did the model find?

Raising the threshold makes the model more cautious: precision usually rises and recall falls. Lowering it does the opposite. A fraud team might accept more false alarms to catch more fraud; a spam filter might prefer to let some spam through rather than lose an important email.

Every threshold gives one point on the ROC curve (receiver operating characteristic), which plots the share of fraud caught against the share of genuine payments flagged. The area under the curve (AUC) summarizes how well the scores separate the classes, whatever threshold you later choose: 0.5 is random guessing, 1.0 is perfect separation.

InteractiveThresholds and the ROC curveDrag the threshold; change the separation

Loading visual…

Slide the threshold left and right to move along the ROC curve. Then make the model worse or better at separating the classes: the curve bends towards the diagonal or the top-left corner, and the AUC follows. With 900 genuine payments and 100 frauds, watch how accuracy stays high even when most fraud is missed.

Training with log loss

To train a classifier we need a loss that compares predicted probabilities with the true labels. The standard choice is log loss, also called binary cross-entropy:

\text{Log loss} = -\frac{1}{n} \sum_{i=1}^{n} \Big[\, y_i \log p_i + (1 - y_i) \log(1 - p_i) \,\Big]

If the true label is 1 and the model says 0.95, the loss is small. If it says 0.05, the loss is large — confident wrong answers are punished hardest. Like mean squared error, log loss is smooth, so gradient descent can minimize it. The “fit” button in the visual does exactly that.

More than two classes

For several categories — ten digits, a hundred product types — logistic regression generalizes with the softmax function, which turns a list of scores into probabilities that add up to 1. You will meet softmax again as the final step of every large language model, choosing the next word.

In this lecture you will