The training pipeline

Pipeline steps

The training pipeline turns fresh data into a validated model. Google's MLOps paper breaks it down into these steps:

1. Data extraction from a database or storage.
2. Data validation: the schema and values are what's expected.
3. Data preparation: cleaning, feature computation, splitting into training and evaluation parts.
4. Training the model.
5. Evaluation: quality metrics on data the model hasn't seen.
6. Model validation: the new model is good enough to deploy.

Each step is a separate task with its own input and output. If training fails, you don't have to extract the data again. Only the failed step reruns.

Train and test

At the preparation step, the data is split into two parts. The model trains on the train set. It's then checked on the test set.

Checking on the same data used for training is pointless: the model might have just memorized it. The test set shows how the model behaves on customers it hasn't seen before.

Data validation

Bad data produces a bad model, and nothing crashes along the way. That's why data is validated before training, and the pipeline stops if something's wrong.

For the churn model, the checks might look like this:

• schema: the user_id, orders_30d, avg_check columns are present, types match;
• values: avg_check isn't negative;
• completeness: no more than 1% of avg_check values are empty;
• distribution: the average order value hasn't tripled in a week.

The last check catches both a broken export and real drift. Either way, a human needs to take a look before the model trains on that data.

There are libraries for these checks, for example Great Expectations, Pandera, and TensorFlow Data Validation.

Model validation

The model is evaluated with quality metrics on the test set. The simplest metric is accuracy: the share of correct answers. The model answered correctly for 870 out of 1000 customers, so accuracy is 0.87.

Per the Google paper, a new model is checked before deployment like this:

• metrics on the test set meet a set threshold;
• metrics are no worse than the current production model's;
• quality is acceptable on individual data slices, for example per region;
• the model is compatible with the infrastructure and the service's API.

Checking by data slice matters because overall accuracy can go up while accuracy for one region drops by half.

Model registry

A model registry is a store of model versions. A container registry holds images with tags; a model registry holds models with version numbers.

Each version has:

• the model file;
• a link to the training run that produced it;
• evaluation metrics.

A version can be given an alias, a named pointer to a version, for example champion for the model running in production. That's the term MLflow uses (lesson 5 covers MLflow). The service doesn't load a specific file. It loads "whatever is currently champion for the churn model." To deploy a new model, you move the alias to the new version. A rollback works the same way: you move the alias back to the old one.

Metadata

For every run, the pipeline records metadata:

• the code version, the commit hash;
• the data version;
• training parameters;
• metrics;
• where the results live: the prepared data, the model.

These records let you answer where model version 14 came from, and reproduce its training. The Google paper calls metadata a prerequisite for reproducibility and pipeline debugging.

What to remember

Training pipeline: extract the data, validate, prepare, train, evaluate, validate the model. The steps are independent and rerun individually.

Data is validated before training. The model is evaluated on a test set it hasn't seen, compared against the current one, and checked by data slice.

The model registry stores model versions, and the alias points to the version in production. Deploying and rolling back just means moving the alias. Each run's metadata makes training reproducible.