A guide to MLOps

ML code vs ML system

Only a small fraction of real-world ML systems is composed of the ML code.

The required surrounding infrastructure is vast and complex.

Difficulties with ML projects

Get out of the context of the experience

I ran the experiment but didn’t get the same results, can you check my way of running the experiment?

Make sure you can build the model at all times

I tried to build the model on my machine but it doesn’t work... Are you sure it builds on yours?

Monitor the evolution of the model over time

I’m not sure my changes really help the model’s performances… I hope it still works in production.

Move to production quickly, efficiently and in a semi-automated way

Is your model available in production? Can I use it with my mobile app/website? How can I do so?

A solution

MLOps

Draw inspiration from Software and DevOps best practices

Adapting these practices to the world of machine learning

Improve the management and quality of machine learning projects

Our proposal

A guide to MLOps

Switch from a Jupyter Notebook to production using state-of-the-art MLOps tools

Go from experience to production on the Cloud

Use the best practices for ML

"Welcome to the team!"

You just have joined a team of data scientists and machine learning (ML) engineers (welcome!).

The team is working on a model capable of visually identifying planets or moons within our solar system from images in a Jupyter Notebook.

The team is facing difficulties to move the model to production.

Your mission is to help the team to improve the model and deploy it to the cloud using MLOps best practices.

The big picture

Target audience

You regularly work with machine learning projects

You want to improve processes to ensure quality

You want to consolidate your current infrastructure

You want to move to the Cloud

Prerequisites

Accessible to everyone!

Basic knowledge of Python and terminal is sufficient

A valid credit card for cloud deployment

You will be accompanied throughout the guide!

Technical checks

Before we start:

macOS, Linux, Windows with WSL2

Editor and Terminal (VS Code recommended)

Python 3.11, pip, git, wget, unzip, docker

GitHub account, Google Cloud account

Access the guide

Access the guide at mlops.swiss-ai-center.ch.

Feel free to open an issue on GitHub if you encounter any difficulties or want to contribute.

Leave us a star if you like the guide!

Clean up

Now that you have completed the guide, it is important to properly manage and remove the resources and environments you have created.

This is necessary to avoid:

• unnecessary incurring costs
• potential security concerns

Conclusion

Congratulations! You have completed the guide to MLOps!

You have learned how to improve the management and quality of machine learning projects.

You are now able to switch from a Jupyter Notebook to production using state-of-the-art MLOps tools.

You can go from experiment to production on the Cloud, using the best practices for ML.

Usual ML workflow

Each member of the team manages their own codebase, their own dataset and their own models.

The reproducibility of the model creation is difficult and cannot be guaranteed over time.

Improvements made to the model are hard to track.

Models are hard to share and deploy in production.

Codebase (1/2)

Current situation

• Each developer has its own codebase
• Sharing the code with peers is difficult

Codebase (2/2)

What we are trying to improve

• Allow each developer to improve a common codebase
• Quickly benefit of the work from others

Data (1/2)

Current situation

• The dataset must be manually downloaded and put in the right place
• Different datasets are being used at the same time
• Datasets are hard to improve

Data (2/2)

What we are trying to improve

• Allow the usage of a common and up-to-date dataset
• Efficiently share new revisions to train the model
• Datasets can be stored anywhere

Reproduce (1/2)

Current situation

• Steps to create the model can be complex
• Intermediate commands should not be skipped
• Hyperparameters are hard to track from one run to another

Reproduce (2/2)

What we are trying to improve

• Document the steps to reproduce the experiment
• Ensure it can be run anytime in the future
• DVC can improve time efficiency

Tracking (1/2)

Current situation

• Changes to a model are difficult to track
• Visualize the differences are hard
• Cannot guarantee the changes are beneficial

Tracking (2/2)

What we are trying to improve

• Have a visual way to identify the consequences of the changes made to a model
• Errors/anomalies are easily identified

Serving and publishing (1/2)

Current situation

• The model is hard to use outside the experiment context
• The model is hard to deploy in production
• The model is hard to share with others

Serving and publishing (2/2)

What we are trying to improve

• The model can be used outside the experiment context
• The model can be deployed in production
• The model can be shared with others

Deployment (1/2)

Current situation

• An experiment can run on one machine but can fail on another
• Models must be prepared to be run outside its experiment context
• Exposing the model to the outside world is hard

Deployment (2/2)

What we are trying to improve

• Run the experiment in a clean state to ensure it works everywhere
• Package the model with all its dependencies
• The model can be used over the Internet by other applications
• Automate the process

Labellisation (1/2)

Current situation

TODO

Labellisation (2/2)

What we are trying to improve

TODO

Sources

• MLOps Venn diagram by Cmbreuel on Wikipedia
• ML system diagram by D. Sculley et. al. NIPS 2015: Hidden technical debt in Machine learning systems
• Robot illustation by OpenClipart-Vectors on Pixabay