Experiment tracking and reproducibility

A data scientist runs a training script forty times in a week: new features, different depths, another learning rate. On Friday the best model is “the one from Tuesday afternoon, I think, with the old features”. Experiment tracking replaces that memory with a record of every run, and reproducibility means any recorded model can be rebuilt exactly — for debugging, for audits, or to retrain it on new data with confidence.

Track every run

An experiment tracker such as MLflow or Weights & Biases stores, for every run: the parameters, the code version, the data version, the metrics and the resulting model artifact. Adding it to a training script takes a few lines.

import mlflow

with mlflow.start_run(run_name="xgb-depth8"):
    mlflow.log_params({"max_depth": 8, "learning_rate": 0.05, "features": "v4", "data": "2025-05-31"})
    model = train(X_train, y_train, max_depth=8, learning_rate=0.05)
    mlflow.log_metrics({"val_auc": evaluate(model, X_val, y_val), "p95_latency_ms": benchmark(model)})
    mlflow.xgboost.log_model(model, "model")          # the artifact, linked to this run

With every run recorded, choosing a model becomes a query instead of an archaeology project. Real decisions involve more than one metric — here, quality and latency.

Experiment tracker

A service that records the parameters, versions, metrics and artifacts of every training run.

Artifact

A file produced by a run — typically the trained model, plus plots or evaluation reports.

Pareto front

The runs that no other run beats on every metric at once.

Model registry

A catalog of approved model versions with their lineage and stage, such as staging or production.

Lineage

The recorded chain from data and code, through a run, to a model version.

Reproducibility: rebuild any model

Could you rebuild the exact model that is serving customers today? Without records, the honest answer is usually no.

In practice: commit code before training and log the commit hash; train from immutable data snapshots (a dated table, or a tool such as DVC or lakeFS) and log their IDs; run inside a container image with pinned library versions; keep configuration in versioned files; and set every random seed. GPU training can still differ in the last decimal places — a tolerance in your comparison is fine, as long as everything else is pinned.

Hyperparameter search

Most model families have hyperparameters that must be tuned: learning rate, depth, regularization. Each trial costs a full training run, so how you search matters. Below, the learning rate matters far more than dropout — which is typical.

• Grid search tries every combination of a few values per parameter. Simple and parallel, but wasteful: most trials repeat the same values of the parameter that matters.
• Random search samples each trial independently. With the same budget it covers the important parameter far better, and it parallelizes just as easily.
• Bayesian optimization fits a model to the results so far and picks the next trial where improvement looks most likely. It needs fewer trials, but trials must run partly in sequence.

A reproducible, well-chosen model is ready to leave the lab. The next lecture covers model serving: how predictions reach users, and how to roll out a new model without risking the business.