Probability and Bayes’ theorem
Machine learning is reasoning under uncertainty. Build intuition for conditional probability with natural frequencies, meet the distributions that model clicks, requests and noise, and see how Bayes’ theorem turns a prior belief and new evidence into an updated belief.
In this lecture you will
- Work with conditional probabilities and avoid the base-rate fallacy
- Apply Bayes’ theorem to real alerts, tests and classifiers
- Recognize the normal, binomial, Poisson, exponential and uniform distributions and when they apply
- Update a belief with data using a prior, a likelihood and a posterior
Models rarely know things for certain. A classifier outputs a probability; a forecast has an error range; an A/B test measures a noisy difference. Probability is the language of this uncertainty, and Bayes’ theorem is its most useful rule: it tells you how to update what you believe when new evidence arrives.
Conditional probability and the base-rate trap
P(A | B) — “the probability of A given B” — is the share of cases with B in which A also happens. The trap is to confuse it with P(B | A). An intrusion detector that catches 95% of attacks has P(alert | attack) = 0.95. But the question the security team cares about is the reverse: when an alert fires, what is P(attack | alert)?
Each square is one of 1,000 logins. Try an attack share of 0.1%: even an excellent detector then produces mostly false alarms. Lower the false-alarm rate to see what really improves the alerts.
Counting cases, as above, is the most reliable way to reason. Written as a formula, it is Bayes’ theorem:
The base rate P(attack) sits in the numerator. When the thing you look for is rare, even small false-positive rates swamp the true positives. The same arithmetic governs medical tests, fraud detection and every classifier applied to rare classes.
- Conditional probability
- P(A | B): the probability of A among the cases where B is true.
- Base rate
- How common something is before any evidence: the prior probability.
- Likelihood
- How probable the observed data are under a particular hypothesis or parameter value.
- Posterior
- The updated probability after combining the prior with the likelihood of the data.
- Distribution
- A description of how likely each possible value of a random quantity is.
Distributions worth knowing
A distribution describes how likely each value of a random quantity is. A few families describe most of what engineers measure — pick the one whose story matches your data.
The bars or curve show the theory; the shaded histogram shows 1,000 random samples. Draw new samples and watch the sample mean and variance wobble around the theoretical values.
- Normal — sums of many small effects: measurement noise, errors, heights. Its famous bell curve also initializes network weights.
- Binomial — the number of successes in n independent tries: clicks from impressions, failed requests out of a batch.
- Poisson — counts of independent events in an interval: requests per second, tickets per day. Its mean equals its variance.
- Exponential — the waiting time between those events. Short waits are common, long ones rare.
- Uniform — every value in a range equally likely: random search over hyperparameters, jitter in retry delays.
Updating beliefs with data
Bayes’ theorem also works with whole distributions. Suppose you want the click-through rate of a new banner. Past campaigns suggest around 20% — that is your prior. As visitors arrive, the likelihood says which rates explain the observed clicks, and the posterior combines the two.
- The priorBefore the test, previous campaigns suggest a click rate of about 20%, with a lot of uncertainty.
- The first dataThe first ten visitors arrive, and three of them click.
- The likelihoodFor each possible rate, how probable is 3 clicks in 10? Rates near 30% explain it best; very low or very high rates explain it poorly.
- The posteriorMultiplying prior by likelihood and rescaling gives the posterior. The belief moves from 20% towards 30% — it sits between the two.
- More dataAfter 100 visitors with 30 clicks, the posterior is much narrower and centered near 29%. The shaded band holds 90% of the belief. The prior now barely matters.
from scipy import stats
prior_a, prior_b = 2, 8 # Beta prior: about 20%
clicks, visitors = 30, 100
posterior = stats.beta(prior_a + clicks, prior_b + visitors - clicks)
print(posterior.mean()) # ≈ 0.29
print(posterior.interval(0.90)) # 90% credible intervalThis completes the core toolkit: vectors, matrices, eigenvectors, gradients and probability. The next module goes deeper — least squares, the singular value decomposition, automatic differentiation and maximum likelihood — and the final module assembles everything into the complete mathematics of a neural network.
Key takeaways
- P(A | B) is the probability of A among the cases where B happened; P(A | B) and P(B | A) are usually very different.
- Bayes’ theorem, P(A | B) = P(B | A) P(A) / P(B), lets you reverse a conditional probability — the base rate P(A) matters enormously.
- A handful of distributions model most real quantities; matching the distribution to the data is the first step of any statistical model.
- Bayesian updating combines a prior with the likelihood of the data; with more data, the posterior narrows and the prior matters less.
Check your understanding
4 questions · answers explainedQ1A detector catches 95% of attacks and flags 5% of normal logins. Attacks are 2% of logins. Roughly what share of alerts are real attacks?
Q2Which distribution usually models the number of requests a server receives per second?
Q3In Bayesian updating, what happens to the posterior as more data arrive?
Q4A spam filter reports P(spam | words). Which quantity did Naive Bayes learn directly from training data?