Monitoring and drift

On launch day a model is as good as it will ever be. After that, customers change, products change, fraudsters adapt and upstream systems get “improved”. None of this raises an error: the model keeps returning confident predictions that are slowly getting worse. Monitoring is how you notice — ideally before your customers do.

What to monitor

Good monitoring is layered, from the cheapest and fastest signals to the slowest and most meaningful:

1. Service health — traffic, error rate, latency. The same as for any API.
2. Data quality — schema, missing values, ranges: the expectations from lecture 2, now run on live requests.
3. Drift — how today’s inputs and predictions compare with the training data.
4. Model quality — accuracy, AUC or error against real outcomes, once the labels arrive.
5. Business outcome — the metric the model was built to move, such as churn, fraud losses or conversion.

The first three are available immediately; the last two arrive late but tell you what actually matters.

Measuring drift

The most common drift measure in industry is the population stability index (PSI). Split the training distribution of a feature into ten equal-sized bins, see what share of today’s values falls into each, and add up how far each share moved:

where e is the expected share (10% per bin) and a the actual share in production. A common rule of thumb: below 0.1 stable, 0.1–0.25 moderate, above 0.25 significant. Statistical tests such as Kolmogorov–Smirnov answer a different question — is there any difference at all? — and with enough data the answer is almost always yes.

import numpy as np

def psi(train, live, bins=10):
    edges = np.quantile(train, np.linspace(0, 1, bins + 1)[1:-1])      # deciles of the training data
    expected = np.bincount(np.searchsorted(edges, train), minlength=bins) / len(train)
    actual = np.bincount(np.searchsorted(edges, live), minlength=bins) / len(live)
    actual = np.clip(actual, 1e-4, None)
    return float(np.sum((actual - expected) * np.log(actual / expected)))

Data drift

A change in the distribution of the model’s inputs, also called covariate shift.

Prediction drift

A change in the distribution of the model’s outputs — for example, far more customers flagged than usual.

Concept drift

A change in the relationship between inputs and the outcome, so the same inputs now mean something different.

PSI

Population stability index: a measure of how much a distribution has shifted, compared bin by bin with a reference.

Label delay

The time between a prediction and the moment its true outcome becomes known.

Data drift versus concept drift

Not every drift hurts, and not every harmful change shows up as drift. The difference matters for what you monitor and how you respond.

Alerts that people trust

Every alert is a trade-off. Thresholds that are too sensitive fire on normal noise, and a team that is paged for nothing soon ignores the pager. Thresholds that are too loose let real incidents run for weeks. Use the dashboard below to find the balance.

When an alert fires, respond in order of likelihood and cost:

• Check the data first. Most “drift” is a broken pipeline: a renamed column, a new code, a unit change. Fix it at the source.
• Roll back if a recent model or code release caused the problem.
• Retrain on recent data when the world has genuinely changed — on a schedule, or triggered by monitoring. Compare the new model with the current one (a champion–challenger evaluation) before promoting it.
• Rethink the features or the model when retraining keeps losing ground.

This completes the first module. You can now take a model from a notebook to a monitored production system: reliable data pipelines, tracked and reproducible training, safe serving and rollouts, and monitoring that closes the loop. The next module builds the platform that makes this repeatable across many models — CI/CD for machine learning, feature stores, model registries and training orchestration.