MLOps maturity levels

Where the levels come from

In the paper "MLOps: Continuous delivery and automation pipelines in machine learning," Google Cloud splits MLOps into three maturity levels, from 0 to 2. The level shows how much of the model's path to production is automated.

Let's walk through the levels using the churn model from the previous lesson. Every morning it gives marketing a list of customers who should get a discount.

Level 0: everything by hand

A data scientist exports data from the database, trains the model in a Jupyter notebook, and sends the file churn_v3.pkl to a backend developer. The developer puts the file into an image and deploys the service.

Signs of level 0, per the Google paper:

• every step, from data preparation to training, is done by hand or with separate scripts;
• the data scientist and the production engineers work separately, and the model is handed over as a file;
• new model versions come out rarely, a few times a year per the paper;
• there's no CI or CD for the model;
• only the inference service is deployed to production, with no training pipeline;
• model quality in production isn't tracked.

What breaks. Six months later nobody remembers what data churn_v3.pkl was trained on, so training can't be reproduced. Meanwhile the model picks up drift, and people find out when discounts go to the wrong customers.

The Google paper notes that level 0 is common at companies that are just starting to apply ML.

Level 1: the model retrains itself

At level 1 the training pipeline is automated. The steps from the notebook become separate tasks that run without a human: extract the data, validate it, compute features, train, evaluate, save the model.

This gives you continuous training (CT), automatic retraining of the model in production. Regular CI/CD has no such step. Per the Google paper, the pipeline is triggered:

• manually;
• on a schedule;
• when new training data arrives;
• when model quality drops;
• when the data distribution shifts noticeably.

The main difference from level 0: you deploy the pipeline to production, and it produces the models itself.

What gets added to the infrastructure:

• data validation before training: the schema and values are what's expected;
• model validation after training: the new model is no worse than the current one;
• a metadata store: which run, on which data, with which parameters, produced which model;
• a feature store, a shared store of features for training and production. The paper lists it as an optional component.

What breaks. The pipeline's own code is still changed and deployed by hand. The Google paper calls level 1 a good fit when new models come from new data rather than new ideas. A new idea, say a different set of features, still needs a manual edit and deploy of the pipeline.

Level 2: CI/CD for the pipeline

At level 2, regular CI/CD applies to the pipeline's code. This gives two loops.

1. Pipeline CI/CD. A merge request changing feature computation runs through tests, the step images are built, and a new pipeline version is deployed.
2. Model CT. The deployed pipeline trains a model on a trigger, validates it, and puts it in the model registry, a store of model versions. From there the model gets deployed to the service.

Tests the Google paper lists for CI:

• unit tests for the feature-computation code;
• unit tests for methods implemented in the model;
• training converges, meaning the model's error shrinks step by step;
• training doesn't produce NaN ("not a number"), from division by zero and similar issues;
• every pipeline step produces the expected artifacts;
• the pipeline steps work correctly together.

Before deploying a model, the paper suggests checking its compatibility with the infrastructure and load-testing the inference service: how many requests per second it can handle and at what latency, meaning response time.

Per the paper, level 2 is needed when you have to update pipelines in production quickly and reliably: trying new ideas often and shipping them.

Figuring out your level

The paper's signs boil down to three questions.

1. Does training run without a human, on a schedule or a trigger?
2. Are the data and the new model validated automatically, before deployment?
3. Does a change to the pipeline's code go through tests and CI/CD?

No to the first question means level 0. Yes to the first two means level 1. Yes to all three means level 2.

What to remember

Level 0: the model is trained by hand and handed over as a file, and quality in production isn't tracked.

Level 1: the training pipeline is automated, and the model retrains on a trigger (continuous training). Data and the model are validated automatically.

Level 2: the pipeline's code goes through CI/CD with tests. You need it when you often ship new ideas, not just new data.

Source: MLOps: Continuous delivery and automation pipelines in machine learning, Google Cloud.