Skip to content

Chapter 2.4 - Track model evolution in the CI/CD pipeline with CML

Introduction

At this point, you have a CI/CD pipeline that will run the experiment on each commit. However, you may want to visualize the results of the experiment in the pipeline. For example, you may want to see the metrics and plots generated by the experiment. This is where CML comes in.

In this chapter, you will learn how to:

  1. Update the CI/CD pipeline configuration file to visualize them with CML
  2. Push the CI/CD pipeline configuration file to Git
  3. Visualize the execution of the CI/CD pipeline

The following diagram illustrates the control flow of the experiment at the end of this chapter:

flowchart TB
    dot_dvc[(.dvc)] <-->|dvc push
                         dvc pull| s3_storage[(S3 Storage)]
    dot_git[(.git)] <-->|git push
                         git pull| gitGraph[Git Remote]
    workspaceGraph <-....-> dot_git
    data[data/raw] <-.-> dot_dvc
    subgraph remoteGraph[REMOTE]
        s3_storage
        subgraph gitGraph[Git Remote]
            direction TB
            repository[(Repository)] --> action[Action]
            action -->|dvc pull| action_data[data/raw]
            action_data -->|dvc repro| action_out[metrics & plots]
            action_out -->|cml publish| pr[Pull Request]
            pr --> repository
        end
    end
    subgraph cacheGraph[CACHE]
        dot_dvc
        dot_git
    end
    subgraph workspaceGraph[WORKSPACE]
        prepare[prepare.py] <-.-> dot_dvc
        train[train.py] <-.-> dot_dvc
        evaluate[evaluate.py] <-.-> dot_dvc
        data --> prepare
        subgraph dvcGraph["dvc.yaml (dvc repro)"]
            prepare --> train
            train --> evaluate
        end
        params[params.yaml] -.- prepare
        params -.- train
        params <-.-> dot_dvc
    end
    style workspaceGraph opacity:0.4,color:#7f7f7f80
    style dvcGraph opacity:0.4,color:#7f7f7f80
    style cacheGraph opacity:0.4,color:#7f7f7f80
    style data opacity:0.4,color:#7f7f7f80
    style dot_git opacity:0.4,color:#7f7f7f80
    style dot_dvc opacity:0.4,color:#7f7f7f80
    style prepare opacity:0.4,color:#7f7f7f80
    style train opacity:0.4,color:#7f7f7f80
    style evaluate opacity:0.4,color:#7f7f7f80
    style params opacity:0.4,color:#7f7f7f80
    style s3_storage opacity:0.4,color:#7f7f7f80
    style repository opacity:0.4,color:#7f7f7f80
    style action opacity:0.4,color:#7f7f7f80
    style action_data opacity:0.4,color:#7f7f7f80
    style action_out opacity:0.4,color:#7f7f7f80
    linkStyle 0 opacity:0.4,color:#7f7f7f80
    linkStyle 1 opacity:0.4,color:#7f7f7f80
    linkStyle 2 opacity:0.4,color:#7f7f7f80
    linkStyle 3 opacity:0.4,color:#7f7f7f80
    linkStyle 4 opacity:0.4,color:#7f7f7f80
    linkStyle 5 opacity:0.4,color:#7f7f7f80
    linkStyle 6 opacity:0.4,color:#7f7f7f80
    linkStyle 8 opacity:0.4,color:#7f7f7f80
    linkStyle 9 opacity:0.4,color:#7f7f7f80
    linkStyle 10 opacity:0.4,color:#7f7f7f80
    linkStyle 11 opacity:0.4,color:#7f7f7f80
    linkStyle 12 opacity:0.4,color:#7f7f7f80
    linkStyle 13 opacity:0.4,color:#7f7f7f80
    linkStyle 14 opacity:0.4,color:#7f7f7f80
    linkStyle 15 opacity:0.4,color:#7f7f7f80
    linkStyle 16 opacity:0.4,color:#7f7f7f80
    linkStyle 17 opacity:0.4,color:#7f7f7f80

Steps

The reports produced by CML compare the current run with a designated target reference.

The target reference can be a specific commit, allowing for a comparison between the current run and the run associated with that particular commit. Alternatively, it can be a branch, enabling a comparison between the current run and the run linked to the target branch.

Numerous workflows facilitate discussions and the integration of work into a target reference. In this guide, you will focus on two methods that are commonly used on GitHub - pull requests (PRs) - and GitLab - merge requests (MRs) - to incorporate the work performed into the main branch.

Update the CI/CD pipeline configuration file

You will enhance the CI/CD pipeline by adding an automated report comparing new parameters and new metrics to the main branch, and published as a comment.

These additions will enable a comprehensive analysis of branches and facilitate collaboration and decision-making within the team.

Update the .github/workflows/mlops.yaml file with the following content. Explore this file to understand the train-and-report stage and its steps:

.github/workflows/mlops.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
name: MLOps

on:
  # Runs on pushes targeting main branch
  push:
    branches:
      - main

  # Runs on pull requests
  pull_request:

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

jobs:
  train-and-report:
    permissions: write-all
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
          cache: pip
      - name: Install dependencies
        run: pip install --requirement requirements-freeze.txt
      - name: Login to Google Cloud
        uses: google-github-actions/auth@v2
        with:
          credentials_json: '${{ secrets.GOOGLE_SERVICE_ACCOUNT_KEY }}'
      - name: Train model
        run: dvc repro --pull
      - name: Setup CML
        if: github.event_name == 'pull_request'
        uses: iterative/setup-cml@v2
        with:
          version: '0.20.0'
      - name: Create CML report
        if: github.event_name == 'pull_request'
        env:
          REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          # Fetch all other Git branches
          git fetch --depth=1 origin main:main

          # Add title to the report
          echo "# Experiment Report (${{ github.sha }})" >> report.md

          # Compare parameters to main branch
          echo "## Params workflow vs. main" >> report.md
          dvc params diff main --md >> report.md

          # Compare metrics to main branch
          echo "## Metrics workflow vs. main" >> report.md
          dvc metrics diff main --md >> report.md

          # Compare plots (images) to main branch
          dvc plots diff main

          # Create plots
          echo "## Plots" >> report.md

          # Create training history plot
          echo "### Training History" >> report.md
          echo "#### main" >> report.md
          echo '![](./dvc_plots/static/main_evaluation_plots_training_history.png "Training History")' >> report.md
          echo "#### workspace" >> report.md
          echo '![](./dvc_plots/static/workspace_evaluation_plots_training_history.png "Training History")' >> report.md

          # Create predictions preview
          echo "### Predictions Preview" >> report.md
          echo "#### main" >> report.md
          echo '![](./dvc_plots/static/main_evaluation_plots_pred_preview.png "Predictions Preview")' >> report.md
          echo "#### workspace" >> report.md
          echo '![](./dvc_plots/static/workspace_evaluation_plots_pred_preview.png "Predictions Preview")' >> report.md

          # Create confusion matrix
          echo "### Confusion Matrix" >> report.md
          echo "#### main" >> report.md
          echo '![](./dvc_plots/static/main_evaluation_plots_confusion_matrix.png "Confusion Matrix")' >> report.md
          echo "#### workspace" >> report.md
          echo '![](./dvc_plots/static/workspace_evaluation_plots_confusion_matrix.png "Confusion Matrix")' >> report.md

          # Publish the CML report
          cml comment update --target=pr --publish report.md

The updated train-and-report job is responsible for reporting the results of the model evaluation and comparing it with the main branch. Some steps in this job are triggered only on pull requests. The job checks out the repository, sets up DVC and CML, creates and publishes the report as a pull request comment.

Check the differences with Git to validate the changes:

Execute the following command(s) in a terminal
1
2
# Show the differences with Git
git diff .github/workflows/mlops.yaml

The output should be similar to this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
diff --git a/.github/workflows/mlops.yaml b/.github/workflows/mlops.yaml
index 5aae2a1..1fa989b 100644
--- a/.github/workflows/mlops.yaml
+++ b/.github/workflows/mlops.yaml
@@ -13,7 +13,8 @@ on:
   workflow_dispatch:

 jobs:
-  train:
+  train-and-report:
+    permissions: write-all
     runs-on: ubuntu-latest
     steps:
       - name: Checkout repository
@@ -31,3 +32,56 @@ jobs:
           credentials_json: '${{ secrets.GOOGLE_SERVICE_ACCOUNT_KEY }}'
       - name: Train model
         run: dvc repro --pull
+      - name: Setup CML
+        if: github.event_name == 'pull_request'
+        uses: iterative/setup-cml@v2
+        with:
+          version: '0.20.0'
+      - name: Create CML report
+        if: github.event_name == 'pull_request'
+        env:
+          REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          # Fetch all other Git branches
+          git fetch --depth=1 origin main:main
+
+          # Add title to the report
+          echo "# Experiment Report (${{ github.sha }})" >> report.md
+
+          # Compare parameters to main branch
+          echo "## Params workflow vs. main" >> report.md
+          dvc params diff main --md >> report.md
+
+          # Compare metrics to main branch
+          echo "## Metrics workflow vs. main" >> report.md
+          dvc metrics diff main --md >> report.md
+
+          # Compare plots (images) to main branch
+          dvc plots diff main
+
+          # Create plots
+          echo "## Plots" >> report.md
+
+          # Create training history plot
+          echo "### Training History" >> report.md
+          echo "#### main" >> report.md
+          echo '![](./dvc_plots/static/main_evaluation_plots_training_history.png "Training History")' >> report.md
+          echo "#### workspace" >> report.md
+          echo '![](./dvc_plots/static/workspace_evaluation_plots_training_history.png "Training History")' >> report.md
+
+          # Create predictions preview
+          echo "### Predictions Preview" >> report.md
+          echo "#### main" >> report.md
+          echo '![](./dvc_plots/static/main_evaluation_plots_pred_preview.png "Predictions Preview")' >> report.md
+          echo "#### workspace" >> report.md
+          echo '![](./dvc_plots/static/workspace_evaluation_plots_pred_preview.png "Predictions Preview")' >> report.md
+
+          # Create confusion matrix
+          echo "### Confusion Matrix" >> report.md
+          echo "#### main" >> report.md
+          echo '![](./dvc_plots/static/main_evaluation_plots_confusion_matrix.png "Confusion Matrix")' >> report.md
+          echo "#### workspace" >> report.md
+          echo '![](./dvc_plots/static/workspace_evaluation_plots_confusion_matrix.png "Confusion Matrix")' >> report.md
+
+          # Publish the CML report
+          cml comment update --target=pr --publish report.md

In order to allow commit from the CI and later generate reports with CML, a Personal Access Token (PAT) must be created. A Project or a Group Access Token are not sufficient for the usage of CML's runners that will be used in the next steps.

To create a Personal Access Token, go in your Profile preferences > Access Tokens.

  • Token name: gitlab-ci[bot]
  • Expiration date: None
  • Select scopes: api, read_repository and write_repository

Select Create personal access token to create the token. Copy it. It will be displayed only once.

Store the PAT as a CI/CD variable by going to Settings > CI/CD from the left sidebar of your GitLab project.

Select Variables and select Add variable.

Create a new variable named GITLAB_PAT with the PAT value as its value.

  • Protect variable: Unchecked
  • Mask variable: Checked
  • Expand variable reference: Unchecked

Save the variable by clicking Add variable.

Update the .gitlab-ci.yml file with the following content. Explore this file to understand the report stage and its steps:

.gitlab-ci.yml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
stages:
  - train
  - report

variables:
  # Change pip's cache directory to be inside the project directory since we can
  # only cache local items.
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
  # https://dvc.org/doc/user-guide/troubleshooting?tab=GitLab-CI-CD#git-shallow
  GIT_DEPTH: "0"
  # Set the path to Google Service Account key for DVC - https://dvc.org/doc/command-reference/remote/add#google-cloud-storage
  GOOGLE_APPLICATION_CREDENTIALS: "${CI_PROJECT_DIR}/google-service-account-key.json"
  # Environment variable for CML
  REPO_TOKEN: $GITLAB_PAT

train:
  stage: train
  image: python:3.11
  rules:
    - if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  before_script:
    # Set the Google Service Account key
    - echo "${GOOGLE_SERVICE_ACCOUNT_KEY}" | base64 -d > $GOOGLE_APPLICATION_CREDENTIALS
    # Create the virtual environment for caching
    - python3.11 -m venv .venv
    - source .venv/bin/activate
  script:
    # Install dependencies
    - pip install --requirement requirements-freeze.txt
    # Run the experiment
    - dvc repro --pull
  cache:
    paths:
      # Pip's cache doesn't store the Python packages
      # https://pip.pypa.io/en/stable/reference/pip_install/#caching
      - .cache/pip
      - .venv/

report:
  stage: report
  image: iterativeai/cml:0-dvc3-base1
  needs:
    - train
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  before_script:
    # Set the Google Service Account key
    - echo "${GOOGLE_SERVICE_ACCOUNT_KEY}" | base64 -d > $GOOGLE_APPLICATION_CREDENTIALS
  script:
    - |
      # Fetch the experiment changes
      dvc pull

      # Fetch all other Git branches
      git fetch --depth=1 origin main:main

      # Add title to the report
      echo "# Experiment Report (${CI_COMMIT_SHA})" >> report.md

      # Compare parameters to main branch
      echo "## Params workflow vs. main" >> report.md
      dvc params diff main --md >> report.md

      # Compare metrics to main branch
      echo "## Metrics workflow vs. main" >> report.md
      dvc metrics diff main --md >> report.md

      # Compare plots (images) to main branch
      dvc plots diff main

      # Create plots
      echo "## Plots" >> report.md

      # Create training history plot
      echo "### Training History" >> report.md
      echo "#### main" >> report.md
      echo '![](./dvc_plots/static/main_evaluation_plots_training_history.png "Training History")' >> report.md
      echo "#### workspace" >> report.md
      echo '![](./dvc_plots/static/workspace_evaluation_plots_training_history.png "Training History")' >> report.md

      # Create predictions preview
      echo "### Predictions Preview" >> report.md
      echo "#### main" >> report.md
      echo '![](./dvc_plots/static/main_evaluation_plots_pred_preview.png "Predictions Preview")' >> report.md
      echo "#### workspace" >> report.md
      echo '![](./dvc_plots/static/workspace_evaluation_plots_pred_preview.png "Predictions Preview")' >> report.md

      # Create confusion matrix
      echo "### Confusion Matrix" >> report.md
      echo "#### main" >> report.md
      echo '![](./dvc_plots/static/main_evaluation_plots_confusion_matrix.png "Confusion Matrix")' >> report.md
      echo "#### workspace" >> report.md
      echo '![](./dvc_plots/static/workspace_evaluation_plots_confusion_matrix.png "Confusion Matrix")' >> report.md

      # Publish the CML report
      cml comment update --target=pr --publish report.md

The new report job is responsible for reporting the results of the model evaluation and comparing it with the main branch. This job is triggered only on merge requests. The job checks out the repository, sets up DVC and CML, creates and publishes the report as a merge request comment.

Note

You may notice that the report stage doesn't use the project dependencies. As you do not need to reproduce the experiment, you can use DVC from the iterativeai/cml:0-dvc3-base1 Docker image without the project dependencies. DVC will then retrieve the data stored on the bucket on its own.

Check the differences with Git to validate the changes.

Execute the following command(s) in a terminal
1
2
# Show the differences with Git
git diff .gitlab-ci.yml

The output should be similar to this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index d2445e9..dc7c69c 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,5 +1,6 @@
 stages:
   - train
+  - report

 variables:
   # Change pip's cache directory to be inside the project directory since we can
@@ -9,6 +10,8 @@ variables:
   GIT_DEPTH: "0"
   # Set the path to Google Service Account key for DVC - https://dvc.org/doc/command-reference/remote/add#google-cloud-storage
   GOOGLE_APPLICATION_CREDENTIALS: "${CI_PROJECT_DIR}/google-service-account-key.json"
+  # Environment variable for CML
+  REPO_TOKEN: $GITLAB_PAT

 train:
   stage: train
@@ -33,3 +36,62 @@ train:
       # https://pip.pypa.io/en/stable/reference/pip_install/#caching
       - .cache/pip
       - .venv/
+
+report:
+  stage: report
+  image: iterativeai/cml:0-dvc3-base1
+  needs:
+    - train
+  rules:
+    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
+  before_script:
+    # Set the Google Service Account key
+    - echo "${GOOGLE_SERVICE_ACCOUNT_KEY}" | base64 -d > $GOOGLE_APPLICATION_CREDENTIALS
+  script:
+    - |
+      # Fetch the experiment changes
+      dvc pull
+
+      # Fetch all other Git branches
+      git fetch --depth=1 origin main:main
+
+      # Add title to the report
+      echo "# Experiment Report (${CI_COMMIT_SHA})" >> report.md
+
+      # Compare parameters to main branch
+      echo "## Params workflow vs. main" >> report.md
+      dvc params diff main --md >> report.md
+
+      # Compare metrics to main branch
+      echo "## Metrics workflow vs. main" >> report.md
+      dvc metrics diff main --md >> report.md
+
+      # Compare plots (images) to main branch
+      dvc plots diff main
+
+      # Create plots
+      echo "## Plots" >> report.md
+
+      # Create training history plot
+      echo "### Training History" >> report.md
+      echo "#### main" >> report.md
+      echo '![](./dvc_plots/static/main_evaluation_plots_training_history.png "Training History")' >> report.md
+      echo "#### workspace" >> report.md
+      echo '![](./dvc_plots/static/workspace_evaluation_plots_training_history.png "Training History")' >> report.md
+
+      # Create predictions preview
+      echo "### Predictions Preview" >> report.md
+      echo "#### main" >> report.md
+      echo '![](./dvc_plots/static/main_evaluation_plots_pred_preview.png "Predictions Preview")' >> report.md
+      echo "#### workspace" >> report.md
+      echo '![](./dvc_plots/static/workspace_evaluation_plots_pred_preview.png "Predictions Preview")' >> report.md
+
+      # Create confusion matrix
+      echo "### Confusion Matrix" >> report.md
+      echo "#### main" >> report.md
+      echo '![](./dvc_plots/static/main_evaluation_plots_confusion_matrix.png "Confusion Matrix")' >> report.md
+      echo "#### workspace" >> report.md
+      echo '![](./dvc_plots/static/workspace_evaluation_plots_confusion_matrix.png "Confusion Matrix")' >> report.md
+
+      # Publish the CML report
+      cml comment update --target=pr --publish report.md

Take some time to understand the changes made to the file.

Push the CI/CD pipeline configuration file to Git

Push the CI/CD pipeline configuration file to Git:

Execute the following command(s) in a terminal
1
2
3
4
5
6
7
8
# Add the configuration file
git add .github/workflows/mlops.yaml

# Commit the changes
git commit -m "Add CML reporting to CI/CD pipeline"

# Push the changes
git push

Push the CI/CD pipeline configuration file to Git:

Execute the following command(s) in a terminal
1
2
3
4
5
6
7
8
# Add the configuration file
git add .gitlab-ci.yml

# Commit the changes
git commit -m "Add CML reporting to CI/CD pipeline"

# Push the changes
git push

Check the results

You can see the pipeline running on the Actions page.

You can see the pipeline running on the CI/CD > Pipelines page.

You should see a pipeline running on the main branch. The pipeline should run the same way as in the previous chapter. In the next chapter, you will see how the CML report is generated and published as a comment on a pull request/merge request.

This chapter is done, you can check the summary.

Summary

Congratulations! You now have a CI/CD pipeline that will run and update the experiment results as well as create a report comparing the results with the main branch on a pull request.

In this chapter, you have successfully:

  1. Updated the CI/CD pipeline configuration file to add an automated report comparing new parameters and new metrics to the main branch, and published as a comment
  2. Pushed the CI/CD pipeline configuration file to Git
  3. Checked the results

You fixed some of the previous issues:

  • CI/CD pipeline is triggered on pull requests and reports the results of the experiment

You can now safely continue to the next chapter.

State of the MLOps process

  • Notebook has been transformed into scripts for production
  • Codebase and dataset are versioned
  • Steps used to create the model are documented and can be re-executed
  • Changes done to a model can be visualized with parameters, metrics and plots to identify differences between iterations
  • Codebase can be shared and improved by multiple developers
  • Dataset can be shared among the developers and is placed in the right directory in order to run the experiment
  • Experiment can be executed on a clean machine with the help of a CI/CD pipeline
  • CI/CD pipeline is triggered on pull requests and reports the results of the experiment
  • Changes to model are not thoroughly reviewed and discussed before integration
  • Model may have required artifacts that are forgotten or omitted in saved/loaded state
  • Model cannot be easily used from outside of the experiment context
  • Model requires manual publication to the artifact registry
  • Model is not accessible on the Internet and cannot be used anywhere
  • Model requires manual deployment on the cluster
  • Model cannot be trained on hardware other than the local machine
  • Model cannot be trained on custom hardware for specific use-cases

You will address these issues in the next chapters for improved efficiency and collaboration. Continue the guide to learn how.

Sources

Highly inspired by:

And the following Git repositories: