Our CI/CD Framework for Matillion DPC (Step‑by‑Step) 

Overview

The purpose of this article is to provide a step-by-step guide for implementing an automated CI/CD framework in Matillion Data Productivity Cloud. Our 2-stage approach, Powered by GitHub Actions, allows teams to ensure consistency, auditability, and repeatability in every release. 

Stage 1: Pull Request (Feature → Prod) Triggers CI Validation 

When a developer raises a PR from a feature branch to prod, a CI workflow automatically kicks off inside GitHub Actions that executes the following steps: 

1. Checkout Feature Branch 

The pipeline checks out the developer’s feature branch to analyze and validate the code changes introduced. 

2. Identify Modified Files 

The workflow scans for: 

• All files included in the branch 

• A subset of files modified as part of the PR 
  This ensures that only relevant Matillion jobs are tested and packaged. 

3. Create & Publish Artifact to DEV-STABLE 

An artifact containing the updated jobs is automatically generated and deployed into the DEV‑STABLE environment—a safe validation space. 

4. Automated Quality Checks 

A custom Python script analyzes the job files to detect common issues such as: 

• Hardcoded values 

• Security risks 

• Inconsistent variable usage 

This adds governance and enforces development standards. 

5. Run Functional Tests in DEV-STABLE 

The modified Matillion jobs execute in the DEV‑STABLE environment to confirm that: 

• Jobs don’t break 

• Dependencies resolve correctly 

• Expected behavior matches design 

6. Approval Gate 

If all checks pass, the pipeline pauses and waits for a manual PR approval—a crucial human‑in‑the‑loop control before merging into production. 

Stage 2: Automated Deployment After Merge to Prod 

Once the PR is approved and merged into prod, the CD portion of the workflow handles automated deployment in the following order:

1. Checkout Prod Branch – The workflow pulls the final, merged production code. 
2. Collect All Production Files – Gather a complete file list to prepare the production-ready artifact. 
3. Create & Publish Production Artifact – A new artifact is built and deployed directly into the Production Matillion DPC environment, making the updated jobs live with no manual intervention. 

Summary

This approach automates every stage of the development lifecycle—from version‑controlled updates in GitHub to rapid validation in DEV environments and seamless, error‑free deployments into production. By integrating quality checks, artifact automation, and approval gates, teams can deliver Matillion pipelines faster and with greater confidence.

Connect with the CloudEQS team today to modernize your data engineering operations and unlock higher performance across your Matillion ecosystem.