Chapter 2.2 - Move the ML experiment data to the cloud¶

Introduction¶

At this point, the codebase is made available to team members using Git, but the experiment data itself is not.

Similarly to other version control system, DVC allows for storing the dataset in a remote storage, typically a cloud storage provider, ensuring effective tracking of modifications and smooth maintenance workflow.

This guide will demonstrate the use of a remote Storage Bucket for storing the dataset.

In this chapter, you will learn how to:

1. Create a project on the cloud provider
2. Create a storage bucket on the cloud provider
3. Configure DVC for remote storage
4. Push the data files to DVC
5. Commit the metadata files to Git

The following diagram illustrates the control flow of the experiment at the end of this chapter:

flowchart TB
    dot_dvc[(.dvc)] <-->|dvc push
                         dvc pull| s3_storage[(S3 Storage)]
    dot_git[(.git)] <-->|git push
                         git pull| gitGraph[Git Remote]
    workspaceGraph <-....-> dot_git
    data[data/raw] <-.-> dot_dvc
    subgraph remoteGraph[REMOTE]
    s3_storage
        subgraph gitGraph[Git Remote]
            repository[(Repository)]
        end
    end
    subgraph cacheGraph[CACHE]
        dot_dvc
        dot_git
    end
    subgraph workspaceGraph[WORKSPACE]
    prepare[prepare.py] <-.-> dot_dvc
    train[train.py] <-.-> dot_dvc
    evaluate[evaluate.py] <-.-> dot_dvc
    data --> prepare
    subgraph dvcGraph["dvc.yaml (dvc repro)"]
    prepare --> train
    train --> evaluate
    end
    params[params.yaml] -.- prepare
    params -.- train
    params <-.-> dot_dvc
    end
    style gitGraph opacity:0.4,color:#7f7f7f80
    style repository opacity:0.4,color:#7f7f7f80
    style workspaceGraph opacity:0.4,color:#7f7f7f80
    style dvcGraph opacity:0.4,color:#7f7f7f80
    style cacheGraph opacity:0.4,color:#7f7f7f80
    style dot_git opacity:0.4,color:#7f7f7f80
    style data opacity:0.4,color:#7f7f7f80
    style prepare opacity:0.4,color:#7f7f7f80
    style train opacity:0.4,color:#7f7f7f80
    style evaluate opacity:0.4,color:#7f7f7f80
    style params opacity:0.4,color:#7f7f7f80
    linkStyle 1 opacity:0.4,color:#7f7f7f80
    linkStyle 2 opacity:0.4,color:#7f7f7f80
    linkStyle 3 opacity:0.4,color:#7f7f7f80
    linkStyle 4 opacity:0.4,color:#7f7f7f80
    linkStyle 5 opacity:0.4,color:#7f7f7f80
    linkStyle 6 opacity:0.4,color:#7f7f7f80
    linkStyle 7 opacity:0.4,color:#7f7f7f80
    linkStyle 8 opacity:0.4,color:#7f7f7f80
    linkStyle 9 opacity:0.4,color:#7f7f7f80
    linkStyle 10 opacity:0.4,color:#7f7f7f80
    linkStyle 11 opacity:0.4,color:#7f7f7f80
    linkStyle 12 opacity:0.4,color:#7f7f7f80

Let's get started!

Steps¶

Install and configure the cloud provider CLI¶

Install and configure the cloud provider CLI tool to manage the cloud resources:

1
2

# Initialize and login to Google Cloud
gcloud init

Create a project on a cloud provider¶

This step will create a project on a cloud provider to host the data.

1
2

# Export the project ID
export GCP_PROJECT_ID=<my_project_id>

1
2
3
4
5

# Create a new project
gcloud projects create $GCP_PROJECT_ID

# Select your Google Cloud project
gcloud config set project $GCP_PROJECT_ID

1
2
3
4

# Set authentication for our ML experiment
# https://dvc.org/doc/user-guide/data-management/remote-storage/google-cloud-storage
# https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login
gcloud auth application-default login

Link a billing account to the project¶

Link a billing account to the project to be able to create to create cloud resources:

1
2

# List the billing accounts
gcloud billing accounts list

1
2

# Export the billing account ID
export GCP_BILLING_ACCOUNT_ID=<my_billing_account_id>

1
2
3

# Link the billing account to the project
gcloud billing projects link $GCP_PROJECT_ID \
    --billing-account $GCP_BILLING_ACCOUNT_ID

Create the Storage Bucket on the cloud provider¶

Create the Storage Bucket to store the data with the cloud provider CLI:

1
2

# Export the bucket name
export GCP_BUCKET_NAME=<my_bucket_name>

1
2

# Export the bucket location
export GCP_BUCKET_LOCATION=<my_bucket_location>

1
2
3
4
5

# Create the Google Storage Bucket
gcloud storage buckets create gs://$GCP_BUCKET_NAME \
    --location=$GCP_BUCKET_LOCATION \
    --uniform-bucket-level-access \
    --public-access-prevention

Install the DVC Storage plugin¶

Install the DVC Storage plugin for the cloud provider:

1
2
3
4

tensorflow==2.17.0
matplotlib==3.9.2
pyyaml==6.0.2
dvc[gs]==3.53.2

1
2

# Show the differences with Git
git diff requirements.txt

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

diff --git a/requirements.txt b/requirements.txt
index 0b88f4a..4b8d3d9 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,4 +1,4 @@
 tensorflow==2.17.0
 matplotlib==3.9.2
 pyyaml==6.0.2
-dvc==3.53.2
+dvc[gs]==3.53.2

1
2
3
4
5

# Install the dependencies
pip install --requirement requirements.txt

# Freeze the dependencies
pip freeze --local --all > requirements-freeze.txt

Configure DVC to use the Storage Bucket¶

Configure DVC to use the Storage Bucket on the cloud provider:

1
2

# Add the Google Storage remote bucket
dvc remote add -d data gs://$GCP_BUCKET_NAME/dvcstore

Check the changes¶

Check the changes with Git to ensure all wanted files are here:

1
2
3
4
5

# Add all the available files
git add .

# Check the changes
git status

The output of the git status command should be similar to this:

1
2
3
4
5
6

On branch main
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
    modified:   .dvc/config
    modified:   requirements-freeze.txt
    modified:   requirements.txt

Push the data files to DVC¶

DVC works as Git. Once you want to share the data, you can use dvc push to upload the data and its cache to the storage provider:

1
2

# Upload the experiment data and cache to the remote bucket
dvc push

Commit the changes to Git¶

You can now push the changes to Git so all team members can get the data from DVC as well.

1
2
3
4
5

# Commit the changes
git commit -m "My ML experiment data is shared with DVC"

# Push the changes
git push

Check the results¶

Open the Bucket Storage on the cloud provider and check that the files were hashed and have been uploaded.

Summary¶

Congratulations! You now have a dataset that can be used and shared among the team.

In this chapter, you have successfully:

1. Created a new project on a cloud provider
2. Installed and configured the cloud provider CLI
3. Created the Storage Bucket on the cloud provider
4. Installed the DVC Storage plugin
5. Configured DVC to use the Storage Bucket
6. Updated the gitignore file and adding the experiment data to DVC
7. Pushed the data files to DVC
8. Commit the changes to Git

You fixed some of the previous issues:

• Data no longer needs manual download and is placed in the right directory.

When used by another member of the team, they can easily get a copy of the experiment data from DVC with the following command:

1
2

# Download experiment data from DVC
dvc pull

With the help of DVC, they can also easily reproduce your experiment and, thanks to caching, only the required steps will be executed:

1
2

# Execute the pipeline
dvc repro

You can now safely continue to the next chapter.

State of the MLOps process¶

• Notebook has been transformed into scripts for production
• Codebase and dataset are versioned
• Steps used to create the model are documented and can be re-executed
• Changes done to a model can be visualized with parameters, metrics and plots to identify differences between iterations
• Codebase can be shared and improved by multiple developers
• Dataset can be shared among the developers and is placed in the right directory in order to run the experiment
• Experiment may not be reproducible on other machines
• CI/CD pipeline does not report the results of the experiment
• Changes to model are not thoroughly reviewed and discussed before integration
• Model may have required artifacts that are forgotten or omitted in saved/loaded state
• Model cannot be easily used from outside of the experiment context
• Model requires manual publication to the artifact registry
• Model is not accessible on the Internet and cannot be used anywhere
• Model requires manual deployment on the cluster
• Model cannot be trained on hardware other than the local machine
• Model cannot be trained on custom hardware for specific use-cases

You will address these issues in the next chapters for improved efficiency and collaboration. Continue the guide to learn how.

Sources¶

Highly inspired by:

• Supported storage types - dvc.org
• Install the Google Cloud CLI - cloud.google.com
• Create storage buckets - cloud.google.com
• gcloud storage buckets create - cloud.google.com