Serving and monitoring a model

Batch and online

Serving is how a model delivers answers in production. There are two modes.

Batch: a scheduled job takes all customers, computes the model's answer for each one, and writes the result to a table. That's how the churn model works: marketing opens a ready-made list in the morning. A Kubernetes CronJob or an Airflow task is enough for batch.

Online: the model sits behind an HTTP API and answers every request, for example the moment a customer opens their cart. Latency and scaling matter here, like for any API.

A service in a container

A simple approach to online serving: a small FastAPI service that loads the model from the model registry on startup and serves /predict.

from fastapi import FastAPI
import mlflow
import pandas as pd

app = FastAPI()
model = mlflow.pyfunc.load_model("models:/churn@champion")  # loaded on startup

@app.post("/predict")
def predict(rows: list[dict]):
    scores = model.predict(pd.DataFrame(rows))
    return {"scores": scores.tolist()}

From there it's just like any other service: Dockerfile, Deployment, Service, HPA, a readiness probe.

KServe

KServe is an open source platform for serving models in Kubernetes. In 2025, the CNCF accepted it as an incubating project.

You describe a model with a single InferenceService resource. KServe picks a container that can run models in that format on its own:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: churn
spec:
  predictor:
    model:
      modelFormat:
        name: sklearn                  # model format
      storageUri: s3://models/churn/v15  # where the model file lives

KServe downloads the model, brings up a Pod, and gives you an address for requests.

Triton and vLLM

For models on GPUs there are dedicated inference servers.

NVIDIA Triton Inference Server runs models from PyTorch, ONNX, TensorRT, and other frameworks. It supports dynamic batching: it groups several incoming requests into one batch to process them on the GPU in a single pass.

vLLM serves large language models (LLMs). It's built on PagedAttention, a way to keep intermediate generation data in GPU memory in small blocks so it doesn't take up extra space. vLLM exposes an API compatible with OpenAI's API.

A Pod gets a GPU through resources.limits with the nvidia.com/gpu resource. For that, the NVIDIA device plugin needs to be running on the node.

Gradual rollout

A new model passed validation on the test set, but it can still behave differently in production. That's why it's rolled out gradually.

• Shadow. Requests go to both models, and the user gets the answer from the old one. The new model's answers are only recorded and compared; it has no effect on users.
• Canary. The new model answers a share of requests, say 10%. If the metrics look fine, the share goes up. In KServe this is the canaryTrafficPercent: 10 field under predictor, which works in KServe's serverless mode.
• A/B test. Two models run for different groups of users. You compare a business metric, for example how many customers who got a discount came back. The Google paper mentions an A/B test as a check before a model becomes the main one.

What to monitor

Service metrics are the same as for any API: latency, requests per second, errors, memory, GPU utilization. Prometheus collects them, as in the observability track.

A model breaks without errors in the logs, so these get added on top:

• the distribution of input data: comparing it against the training data catches data drift;
• the distribution of predictions: for example, the model suddenly starts giving a 0.9 churn probability to almost everyone;
• model quality: accuracy against real outcomes.

The last metric comes with a delay. Whether a customer churned or not only shows up several weeks after the prediction. Input and prediction distributions can be checked every day, while accuracy comes later.

There's an open source library called Evidently for finding drift. It compares current data against a reference dataset, for example the training data, and shows which features' distributions have changed. A sharp distribution shift is on the Google paper's list of retraining triggers.

What to remember

Batch computes the model's answers on a schedule; online answers every request. Online serving is done as a service in a container or through KServe. Triton and vLLM are for models on GPU.

A new model is rolled out through shadow, canary, or an A/B test.

Besides service metrics, you monitor the distributions of inputs and predictions, plus accuracy against real outcomes. Drift is detected from distributions, and sharp drift triggers retraining.