Chapter 2.4 - Track model evolution in the CI/CD pipeline with CML¶
Introduction¶
At this point, you have a CI/CD pipeline that will run the experiment on each commit. However, you may want to visualize the results of the experiment in the pipeline. For example, you may want to see the metrics and plots generated by the experiment. This is where CML comes in.
In this chapter, you will learn how to:
- Update the CI/CD pipeline configuration file to visualize them with CML
- Push the CI/CD pipeline configuration file to Git
- Visualize the execution of the CI/CD pipeline
The following diagram illustrates the control flow of the experiment at the end of this chapter:
flowchart TB
dot_dvc[(.dvc)] <-->|dvc push
dvc pull| s3_storage[(S3 Storage)]
dot_git[(.git)] <-->|git push
git pull| gitGraph[Git Remote]
workspaceGraph <-....-> dot_git
data[data/raw] <-.-> dot_dvc
subgraph remoteGraph[REMOTE]
s3_storage
subgraph gitGraph[Git Remote]
direction TB
repository[(Repository)] --> action[Action]
action -->|dvc pull| action_data[data/raw]
action_data -->|dvc repro| action_out[metrics & plots]
action_out -->|cml publish| pr[Pull Request]
pr --> repository
end
end
subgraph cacheGraph[CACHE]
dot_dvc
dot_git
end
subgraph workspaceGraph[WORKSPACE]
prepare[prepare.py] <-.-> dot_dvc
train[train.py] <-.-> dot_dvc
evaluate[evaluate.py] <-.-> dot_dvc
data --> prepare
subgraph dvcGraph["dvc.yaml (dvc repro)"]
prepare --> train
train --> evaluate
end
params[params.yaml] -.- prepare
params -.- train
params <-.-> dot_dvc
end
style workspaceGraph opacity:0.4,color:#7f7f7f80
style dvcGraph opacity:0.4,color:#7f7f7f80
style cacheGraph opacity:0.4,color:#7f7f7f80
style data opacity:0.4,color:#7f7f7f80
style dot_git opacity:0.4,color:#7f7f7f80
style dot_dvc opacity:0.4,color:#7f7f7f80
style prepare opacity:0.4,color:#7f7f7f80
style train opacity:0.4,color:#7f7f7f80
style evaluate opacity:0.4,color:#7f7f7f80
style params opacity:0.4,color:#7f7f7f80
style s3_storage opacity:0.4,color:#7f7f7f80
style repository opacity:0.4,color:#7f7f7f80
style action opacity:0.4,color:#7f7f7f80
style action_data opacity:0.4,color:#7f7f7f80
style action_out opacity:0.4,color:#7f7f7f80
linkStyle 0 opacity:0.4,color:#7f7f7f80
linkStyle 1 opacity:0.4,color:#7f7f7f80
linkStyle 2 opacity:0.4,color:#7f7f7f80
linkStyle 3 opacity:0.4,color:#7f7f7f80
linkStyle 4 opacity:0.4,color:#7f7f7f80
linkStyle 5 opacity:0.4,color:#7f7f7f80
linkStyle 6 opacity:0.4,color:#7f7f7f80
linkStyle 8 opacity:0.4,color:#7f7f7f80
linkStyle 9 opacity:0.4,color:#7f7f7f80
linkStyle 10 opacity:0.4,color:#7f7f7f80
linkStyle 11 opacity:0.4,color:#7f7f7f80
linkStyle 12 opacity:0.4,color:#7f7f7f80
linkStyle 13 opacity:0.4,color:#7f7f7f80
linkStyle 14 opacity:0.4,color:#7f7f7f80
linkStyle 15 opacity:0.4,color:#7f7f7f80
linkStyle 16 opacity:0.4,color:#7f7f7f80
linkStyle 17 opacity:0.4,color:#7f7f7f80Steps¶
The reports produced by CML compare the current run with a designated target reference.
The target reference can be a specific commit, allowing for a comparison between the current run and the run associated with that particular commit. Alternatively, it can be a branch, enabling a comparison between the current run and the run linked to the target branch.
Numerous workflows facilitate discussions and the integration of work into a
target reference. You will focus on a method that is commonly used on GitHub -
pull requests (PRs) - to incorporate the work performed into the main branch.
Update the CI/CD pipeline configuration file¶
You will enhance the CI/CD pipeline by adding an automated report comparing new parameters and new metrics to the main branch, and published as a comment.
These additions will enable a comprehensive analysis of branches and facilitate collaboration and decision-making within the team.
Update the .github/workflows/mlops.yaml file with the following content.
Explore this file to understand the train-and-report stage and its steps:
name: MLOps
on:
# Runs on pushes targeting main branch
push:
branches:
- main
# Runs on pull requests
pull_request:
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:
jobs:
train-and-report:
permissions: write-all
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v7
- name: Setup Python
uses: actions/setup-python@v6
with:
python-version: '3.13'
cache: pip
- name: Install dependencies
run: pip install -r requirements-freeze.txt
- name: Login to Google Cloud
uses: google-github-actions/auth@v3
with:
credentials_json: '${{ secrets.GOOGLE_SERVICE_ACCOUNT_KEY }}'
- name: Train model
run: dvc repro --pull
- name: Setup CML
if: github.event_name == 'pull_request'
uses: iterative/setup-cml@v2
with:
version: '0.20.6'
- name: Create CML report
if: github.event_name == 'pull_request'
env:
REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
# Fetch all other Git branches
git fetch --depth=1 origin main:main
# Add title to the report
echo "# Experiment Report (${{ github.sha }})" >> report.md
# Compare parameters to main branch
echo "## Params workspace vs. main" >> report.md
dvc params diff main --md >> report.md
# Compare metrics to main branch
echo "## Metrics workspace vs. main" >> report.md
dvc metrics diff main --md >> report.md
# Compare plots (images) to main branch
dvc plots diff main
# Create plots
echo "## Plots" >> report.md
# Create training history plot
echo "### Training History" >> report.md
echo "#### main" >> report.md
echo '' >> report.md
echo "#### workspace" >> report.md
echo '' >> report.md
# Create predictions preview
echo "### Predictions Preview" >> report.md
echo "#### main" >> report.md
echo '' >> report.md
echo "#### workspace" >> report.md
echo '' >> report.md
# Create confusion matrix
echo "### Confusion Matrix" >> report.md
echo "#### main" >> report.md
echo '' >> report.md
echo "#### workspace" >> report.md
echo '' >> report.md
# Publish the CML report
cml comment update --target=pr --publish report.md
The updated train-and-report job is responsible for reporting the results of
the model evaluation and comparing it with the main branch. Some steps in this
job are triggered only on pull requests. The job checks out the repository, sets
up DVC and CML, creates and publishes the report as a pull request comment.
Check the differences with Git to validate the changes:
# Show the differences with Git
git diff .github/workflows/mlops.yaml
The output should be similar to this:
diff --git a/.github/workflows/mlops.yaml b/.github/workflows/mlops.yaml
index 5aae2a1..1fa989b 100644
--- a/.github/workflows/mlops.yaml
+++ b/.github/workflows/mlops.yaml
@@ -6,11 +6,15 @@ on:
branches:
- main
+ # Runs on pull requests
+ pull_request:
+
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:
jobs:
- train:
+ train-and-report:
+ permissions: write-all
runs-on: ubuntu-latest
steps:
- name: Checkout repository
@@ -28,4 +32,57 @@ jobs:
credentials_json: '${{ secrets.GOOGLE_SERVICE_ACCOUNT_KEY }}'
- name: Train model
run: dvc repro --pull
+ - name: Setup CML
+ if: github.event_name == 'pull_request'
+ uses: iterative/setup-cml@v2
+ with:
+ version: '0.20.6'
+ - name: Create CML report
+ if: github.event_name == 'pull_request'
+ env:
+ REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ run: |
+ # Fetch all other Git branches
+ git fetch --depth=1 origin main:main
+
+ # Add title to the report
+ echo "# Experiment Report (${{ github.sha }})" >> report.md
+
+ # Compare parameters to main branch
+ echo "## Params workspace vs. main" >> report.md
+ dvc params diff main --md >> report.md
+
+ # Compare metrics to main branch
+ echo "## Metrics workspace vs. main" >> report.md
+ dvc metrics diff main --md >> report.md
+
+ # Compare plots (images) to main branch
+ dvc plots diff main
+
+ # Create plots
+ echo "## Plots" >> report.md
+
+ # Create training history plot
+ echo "### Training History" >> report.md
+ echo "#### main" >> report.md
+ echo '' >> report.md
+ echo "#### workspace" >> report.md
+ echo '' >> report.md
+
+ # Create predictions preview
+ echo "### Predictions Preview" >> report.md
+ echo "#### main" >> report.md
+ echo '' >> report.md
+ echo "#### workspace" >> report.md
+ echo '' >> report.md
+
+ # Create confusion matrix
+ echo "### Confusion Matrix" >> report.md
+ echo "#### main" >> report.md
+ echo '' >> report.md
+ echo "#### workspace" >> report.md
+ echo '' >> report.md
+
+ # Publish the CML report
+ cml comment update --target=pr --publish report.md
Take some time to understand the changes made to the file.
Push the CI/CD pipeline configuration file to Git¶
Push the CI/CD pipeline configuration file to Git:
# Add the configuration file
git add .github/workflows/mlops.yaml
# Commit the changes
git commit -m "Add CML reporting to CI/CD pipeline"
# Push the changes
git push
Check the results¶
You can see the pipeline running on the Actions page.
You should see a pipeline running on the main branch. The pipeline should run
the same way as in the previous chapter. In the next chapter, you will see how
the CML report is generated and published as a comment on a pull request/merge
request.
This chapter is done, you can check the summary.
Summary¶
Congratulations! You now have a CI/CD pipeline that will run and update the experiment results as well as create a report comparing the results with the main branch on a pull request.
In this chapter, you have successfully:
- Updated the CI/CD pipeline configuration file to add an automated report comparing new parameters and new metrics to the main branch, and published as a comment
- Pushed the CI/CD pipeline configuration file to Git
- Checked the results
You fixed some of the previous issues:
- CI/CD pipeline is triggered on pull requests and reports the results of the experiment
Take away
- CML brings experiment results into code review: By automatically posting comparison reports to pull requests, CML makes model performance changes visible alongside code changes, enabling informed decisions before merging.
- Automated reporting reduces manual effort: Instead of manually running
dvc metrics diffand sharing screenshots, CML generates comprehensive reports with parameters, metrics, and plots automatically on every PR. - Visual comparisons tell the story: Side-by-side plots showing training history, predictions, and confusion matrices from main vs. workspace make it immediately clear whether changes improve or degrade model performance.
- Integration with the PR workflow encourages best practices: By tying experiment reports to pull requests, CML naturally encourages code review, discussion, and validation before changes are merged into the main branch.
State of the MLOps process¶
- Codebase can be shared and improved by multiple developers
- Dataset can be shared among the developers and is placed in the right directory in order to run the experiment
- Experiment can be executed on a clean machine with the help of a CI/CD pipeline
- CI/CD pipeline is triggered on pull requests and reports the results of the experiment
- Changes to model are not thoroughly reviewed and discussed before integration
Continue to the next chapters to address the remaining items.
Sources¶
Highly inspired by:
And the following Git repositories: