Model serving

A trained model is a file. It creates value only when its predictions reach a decision: an offer in an app, a flagged payment, a ticket routed to the right team. Model serving is how predictions get there — reliably, fast enough, and without breaking anything when the model changes.

Batch, online or streaming?

There are three basic ways to deliver predictions. The right one depends on how fresh a prediction must be when it is used.

Packaging a model as a service

For online serving, the model is wrapped in a small web service with a versioned API and a strict input schema, packaged as a container, and deployed like any other service. Every response should say which model version produced it, and every prediction should be logged with its inputs — you will need them for monitoring and debugging.

from fastapi import FastAPI
from pydantic import BaseModel
import joblib

app = FastAPI()
model = joblib.load("churn-model-v7.joblib")         # pulled from the model registry at build time

class Customer(BaseModel):
    orders_30d: int
    avg_basket_usd: float
    days_since_last_order: float

@app.post("/v1/churn")
def predict(c: Customer):
    p = model.predict_proba([[c.orders_30d, c.avg_basket_usd, c.days_since_last_order]])[0, 1]
    return {"churn_probability": round(float(p), 4), "model_version": "v7"}

Model server

A service that loads a model and answers prediction requests, such as a FastAPI app, KServe, Triton or a managed endpoint.

Shadow deployment

Sending a copy of live traffic to a new model and logging its predictions without using them.

Canary release

Sending a small, growing share of real traffic to a new version while comparing it with the stable one.

Rollback

Returning all traffic to the previous version, ideally automatically.

SLO

Service-level objective: a measurable target such as “99% of predictions within 150 ms”.

Rolling out a new model safely

A model that scored better offline can still fail in production: a missing feature, a new library, a slower instance type. Safe rollouts limit the blast radius. A shadow deployment first runs the new model on copied traffic with no effect on users. A canary release then sends it a small share of real traffic, compares it with the stable model on guardrail metrics — errors, latency, business KPIs — and increases the share step by step.

Notice that at 5% traffic the canary’s numbers are noisy: small samples need time before a decision is trustworthy. Real rollout tools wait for a minimum number of requests at each step, and compare versions with a statistical test rather than a single threshold.

Where the milliseconds go

Online predictions usually have a latency objective, stated for the tail — for example, 99% of requests within 150 ms. Break a request into its stages and see which choices blow the budget.

The biggest levers are usually the model (smaller, distilled or quantized models, or a GPU), the feature source (precomputed features in an online store instead of computing them per request) and capacity (enough replicas and autoscaling so requests do not queue at peak).

Once a model is live, the work changes from shipping to watching. The last lecture of this module covers monitoring and drift.