Chapter 2.2 - Move the ML experiment data to the cloud¶

Introduction¶

At this point, the codebase is made available to team members using Git, but the experiment data itself is not.

Similarly to other version control systems, DVC allows for storing the dataset in a remote storage, typically a cloud storage provider, ensuring effective tracking of modifications and smooth maintenance workflow.

This guide will demonstrate the use of a remote Storage Bucket for storing the dataset.

In this chapter, you will learn how to:

Create a project on the cloud provider
Create a Storage Bucket on the cloud provider
Configure DVC for remote storage
Push the data files to DVC
Commit the metadata files to Git

The following diagram illustrates the control flow of the experiment at the end of this chapter:

flowchart TB
    dot_dvc[(.dvc)] <-->|dvc push
                         dvc pull| s3_storage[(S3 Storage)]
    dot_git[(.git)] <-->|git push
                         git pull| gitGraph[Git Remote]
    workspaceGraph <-....-> dot_git
    data[data/raw] <-.-> dot_dvc
    subgraph remoteGraph[REMOTE]
    s3_storage
        subgraph gitGraph[Git Remote]
            repository[(Repository)]
        end
    end
    subgraph cacheGraph[CACHE]
        dot_dvc
        dot_git
    end
    subgraph workspaceGraph[WORKSPACE]
    prepare[prepare.py] <-.-> dot_dvc
    train[train.py] <-.-> dot_dvc
    evaluate[evaluate.py] <-.-> dot_dvc
    data --> prepare
    subgraph dvcGraph["dvc.yaml (dvc repro)"]
    prepare --> train
    train --> evaluate
    end
    params[params.yaml] -.- prepare
    params -.- train
    params <-.-> dot_dvc
    end
    style gitGraph opacity:0.4,color:#7f7f7f80
    style repository opacity:0.4,color:#7f7f7f80
    style workspaceGraph opacity:0.4,color:#7f7f7f80
    style dvcGraph opacity:0.4,color:#7f7f7f80
    style cacheGraph opacity:0.4,color:#7f7f7f80
    style dot_git opacity:0.4,color:#7f7f7f80
    style data opacity:0.4,color:#7f7f7f80
    style prepare opacity:0.4,color:#7f7f7f80
    style train opacity:0.4,color:#7f7f7f80
    style evaluate opacity:0.4,color:#7f7f7f80
    style params opacity:0.4,color:#7f7f7f80
    linkStyle 1 opacity:0.4,color:#7f7f7f80
    linkStyle 2 opacity:0.4,color:#7f7f7f80
    linkStyle 3 opacity:0.4,color:#7f7f7f80
    linkStyle 4 opacity:0.4,color:#7f7f7f80
    linkStyle 5 opacity:0.4,color:#7f7f7f80
    linkStyle 6 opacity:0.4,color:#7f7f7f80
    linkStyle 7 opacity:0.4,color:#7f7f7f80
    linkStyle 8 opacity:0.4,color:#7f7f7f80
    linkStyle 9 opacity:0.4,color:#7f7f7f80
    linkStyle 10 opacity:0.4,color:#7f7f7f80
    linkStyle 11 opacity:0.4,color:#7f7f7f80
    linkStyle 12 opacity:0.4,color:#7f7f7f80

Let's get started!

Steps¶

Install and configure the cloud provider CLI¶

Install and configure the cloud provider CLI tool to manage the cloud resources:

To install gcloud, follow the official documentation: Install the Google Cloud CLI - cloud.google.com

Initialize and configure the Google Cloud CLI

The following process will authenticate to Google Cloud using the Google Cloud CLI with the following command. It should open a browser window to authenticate to Google Cloud. You might need to follow the instructions in the terminal to authenticate:

Warning

If gcloud asks you to pick a project or create a project, exit the process by pressing Ctrl+C in the terminal and follow the next steps to create a project.

Execute the following command(s) in a terminal

# Initialize and login to Google Cloud
gcloud init

Create a project on a cloud provider¶

This step will create a project on a cloud provider to host the data.

Warning

Do not create a new project through the web interface. The following commands will create a new project and link it to a billing account for you, without navigating through the web interface.

Export a Google Cloud Project ID with the following command. Replace <my_project_id> with a project ID of your choice. It has to be lowercase and words separated by hyphens.

Warning

The project ID must be unique across all Google Cloud projects and users. For example, use mlops-<surname>-project, where surname is based on your name. Change the project ID if the command fails.

Execute the following command(s) in a terminal

# Export the project ID
export GCP_PROJECT_ID=<my_project_id>

Create a Google Cloud Project with the following commands:

Execute the following command(s) in a terminal

# Create a new project
gcloud projects create $GCP_PROJECT_ID

# Select your Google Cloud project
gcloud config set project $GCP_PROJECT_ID

Then run the following command to authenticate to Google Cloud with the Application Default Credentials. It will create a credentials file in ~/.config/gcloud/application_default_credentials.json. This file must not be shared and will be used by DVC to authenticate to Google Cloud Storage.

Execute the following command(s) in a terminal

# Set authentication for our ML experiment
# https://dvc.org/doc/user-guide/data-management/remote-storage/google-cloud-storage
# https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login
gcloud auth application-default login

Link a billing account to the project¶

Link a billing account to the project to be able to create cloud resources:

List the billing accounts with the following command:

Execute the following command(s) in a terminal

# List the billing accounts
gcloud billing accounts list

If no billing account is available, you can add a new one from the Google Cloud Console and then link it to the project.

Export the billing ACCOUNT_ID with the following command. Replace <my_billing _account_id> with your own billing ACCOUNT_ID:

Execute the following command(s) in a terminal

# Export the billing ACCOUNT_ID
export GCP_BILLING_ACCOUNT_ID=<my_billing_account_id>

Link a billing account to the project with the following command:

Execute the following command(s) in a terminal

# Link the billing account to the project
gcloud billing projects link $GCP_PROJECT_ID \
    --billing-account $GCP_BILLING_ACCOUNT_ID

Create the Storage Bucket on the cloud provider¶

Create the Storage Bucket to store the data with the cloud provider CLI:

Info

The project must be linked to an active billing account to be able to create the bucket. You must set up a valid billing account.

Create the Google Storage Bucket to store the data with the Google Cloud CLI.

Export the bucket name as an environment variable. Replace <my_bucket_name> with a bucket name of your choice. It has to be lowercase and words separated by hyphens.

Warning

The bucket name must be unique across all Google Cloud projects and users. For example, use mlops-<surname>-bucket, where surname is based on your name. Change the bucket name if the command fails.

Execute the following command(s) in a terminal

# Export the bucket name
export GCP_BUCKET_NAME=<my_bucket_name>

Export the bucket location as an environment variable. You can view the available locations at Cloud locations. You should ideally select a location close to where most of the expected traffic will come from. Replace <my_bucket_location> with your own zone. For example, use europe-west6 for Switzerland (Zurich):

Execute the following command(s) in a terminal

# Export the bucket location
export GCP_BUCKET_LOCATION=<my_bucket_location>

Create the bucket:

Execute the following command(s) in a terminal

# Create the Google Storage Bucket
gcloud storage buckets create gs://$GCP_BUCKET_NAME \
    --location=$GCP_BUCKET_LOCATION \
    --uniform-bucket-level-access \
    --public-access-prevention

You now have everything you need for DVC.

Install the DVC Storage plugin¶

Install the DVC Storage plugin for the cloud provider:

Here, the dvc[gs] package enables support for Google Cloud Storage. Update the requirements.txt file:

requirements.txt

tensorflow==2.21.0
matplotlib==3.11.0
scikit-learn==1.9.0
pyyaml==6.0.3
dvc[gs]==3.67.1

Check the differences with Git to validate the changes:

Execute the following command(s) in a terminal

# Show the differences with Git
git diff requirements.txt

The output should be similar to this:

diff --git a/requirements.txt b/requirements.txt
index 116c388..5f775da 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -2,4 +2,4 @@ tensorflow==2.21.0
 matplotlib==3.11.0
 scikit-learn==1.9.0
 pyyaml==6.0.3
-dvc==3.67.1
+dvc[gs]==3.67.1

Install the dependencies and update the freeze file:

Warning

Prior to running any pip commands, it is crucial to ensure the virtual environment is activated to avoid potential conflicts with system-wide Python packages.

To check its status, simply run which python. If the virtual environment is active, the output will show the path to the virtual environment's Python executable. If it is not, you can activate it with source .venv/bin/activate.

Using pip Using uv

Execute the following command(s) in a terminal

# Install the dependencies
pip install -r requirements.txt

# Freeze the dependencies
pip freeze --local --all > requirements-freeze.txt

Execute the following command(s) in a terminal

# Install the dependencies
uv pip install -r requirements.txt

# Freeze the dependencies
uv pip freeze > requirements-freeze.txt

Configure DVC to use the Storage Bucket¶

Configure DVC to use the Storage Bucket on the cloud provider:

Configure DVC to use a Google Storage remote bucket. The dvcstore is a user-defined path on the bucket. You can change it if needed:

Execute the following command(s) in a terminal

# Add the Google Storage remote bucket
dvc remote add -d data gs://$GCP_BUCKET_NAME/dvcstore

Check the changes¶

Check the changes with Git to ensure all wanted files are here:

Execute the following command(s) in a terminal

# Add all the available files
git add .

# Check the changes
git status

The output of the git status command should be similar to this:

On branch main
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
    modified:   .dvc/config
    modified:   requirements-freeze.txt
    modified:   requirements.txt

Push the data files to DVC¶

DVC works as Git. Once you want to share the data, you can use dvc push to upload the data and its cache to the storage provider:

Execute the following command(s) in a terminal

# Upload the experiment data and cache to the remote bucket
dvc push

Commit the changes to Git¶

You can now push the changes to Git so all team members can get the data from DVC as well.

Execute the following command(s) in a terminal

# Commit the changes
git commit -m "Use DVC to share my ML experiment data"

# Push the changes
git push

Check the results¶

Open the Storage Bucket on the cloud provider and check that the files were hashed and have been uploaded.

Open the Cloud Storage on the Google Cloud interface and click on your bucket to access the details.

Tip

If the DVC data is not visible, check that the correct project is selected in the project selector at the top of the Google Cloud interface.

Summary¶

Congratulations! You now have a dataset that can be used and shared among the team.

In this chapter, you have successfully:

Created a new project on a cloud provider
Installed and configured the cloud provider CLI
Created the Storage Bucket on the cloud provider
Installed the DVC Storage plugin
Configured DVC to use the Storage Bucket
Updated the gitignore file and added the experiment data to DVC
Pushed the data files to DVC
Committed the changes to Git

You fixed some of the previous issues:

Dataset can be shared among the developers and is placed in the right directory in order to run the experiment

When used by another member of the team, they can easily get a copy of the experiment data from DVC with the following command:

Execute the following command(s) in a terminal

# Download experiment data from DVC
dvc pull

With the help of DVC, they can also easily reproduce your experiment and, thanks to caching, only the required steps will be executed:

Execute the following command(s) in a terminal

# Execute the pipeline
dvc repro

Take away

Cloud storage buckets are designed for large data files: Unlike Git, which struggles with large files, cloud storage (S3, Google Cloud Storage, etc.) efficiently handles datasets of any size with built-in redundancy and accessibility.
DVC extends version control to data: By configuring a DVC remote pointing to cloud storage, you get Git-like versioning for your datasets with dvc push and dvc pull commands that mirror Git's workflow.
DVC separates metadata from payload to keep repositories lightweight: The .dvc files and dvc.yaml tracked by Git are small pointers to your data; the actual datasets and artifacts live in cloud storage. This split is what makes versioning multi-gigabyte files practical without bloating your Git repository.
DVC plugins bridge storage providers: Installing the appropriate DVC storage plugin (like dvc[gs] for Google Cloud) allows DVC to communicate with various cloud providers using a unified interface.

State of the MLOps process¶

Codebase can be shared and improved by multiple developers
Dataset can be shared among the developers and is placed in the right directory in order to run the experiment
Experiment may not be reproducible on other machines
CI/CD pipeline does not report the results of the experiment
Changes to model are not thoroughly reviewed and discussed before integration

Continue to the next chapters to address the remaining items.

Sources¶

Highly inspired by: