Data and feature pipelines

Ask ML teams where their production incidents come from and the answer is rarely “the model”. It is a column that changed type, a feed that arrived late, a partner that switched units, or a feature computed one way in training and another way in production. This lecture is about the pipelines that turn raw data into model inputs — and the checks that keep them honest.

From source systems to a feature store

A feature is a model input such as “orders in the last 30 days”. A feature pipeline computes features from raw data on a schedule and stores them where both training and serving can read them. Follow a daily batch through one.

Feature

A model input computed from raw data, such as average basket value over 30 days.

Feature store

A system that stores feature values — historical for training, latest for serving — from one shared definition.

Data validation

Automated checks on schema, ranges, distributions, freshness and volume before data is used.

Training–serving skew

A mismatch between the features a model saw in training and those it receives in production.

Point-in-time join

Joining each training example with feature values as they were at that example’s timestamp, never later.

Validate every batch

Data validation works like unit tests for data. You write expectations — the customer ID is never missing, amounts stay near their usual range, the newest record is less than a day old — and run them on every batch. Inject some real-world problems below.

Good expectations come in five families: schema (columns and types), validity (ranges and allowed values), distribution (means, quantiles, category shares compared with a reference), freshness and volume. Decide for each whether a failure should stop the pipeline or only warn — stopping on everything creates alert fatigue, and warning on everything means nothing is ever stopped.

import pandera as pa

schema = pa.DataFrameSchema({
    "customer_id": pa.Column(str, nullable=False, unique=True),
    "country": pa.Column(str, pa.Check.isin(["IN", "US", "GB", "DE", "SG"])),
    "age": pa.Column(int, pa.Check.in_range(18, 100)),
    "amount_usd": pa.Column(float, pa.Check(lambda s: abs(s.mean() / 41.9 - 1) < 0.2)),
})
schema.validate(batch)          # raises an error describing every failed check

Training–serving skew

A churn model uses one feature: days since the last order. The training pipeline computes it in Python; the production service was rewritten in Java. Each implementation looks right on its own. Compare what the model sees — and what happens to the business.

The “missing → 0” case is the dangerous one: nothing crashes, overall accuracy barely changes, and nobody notices that the customers most likely to leave are no longer flagged. The reliable fix is structural — one feature definition, computed by one pipeline and read by both training and serving, which is the main job of a feature store. Logging the features the model receives in production, and comparing their distribution with training, catches whatever slips through.

With reliable data in place, the next step is to make training itself reliable: experiment tracking and reproducibility.