Project Summary
CloudEQS recently migrated a Network Security customer from Matillion’s legacy ETL platform, Matillion ETL, to their AI-enabled platform, MAIA.
- 664 Matillion jobs migrated
- 46 Production schedules cut over
- 17 Weeks from kick-off to go-live
- 9 Core data pipelines re-platformed
The Challenge
Migrating a live, business-critical ETL environment is never just a lift-and-shift. The team needed to:
- Stand up an enterprise-ready environment on the new platform — credentials, role-based access control, API profiles, JDBC drivers, and environment variables all had to be rebuilt from scratch.
- Migrate hundreds of production jobs spanning 9+ core business systems (Salesforce, Hive, BigQuery, Google Sheets, NetSuite, MongoDB, and ServiceNow) without disrupting the schedules the business depended on.
- Recreate CI/CD automation equivalent to — or better than — what the team had on the legacy platform.
- Do all of this while fully enabling the customer’s own Development and Data Ops teams to work independently on the new platform once the migration was complete.
All whilst minimizing impact on live, production systems.
The Approach
As experts in Change Management, the CloudEQS team followed their Matillion Migration Framework. If you’re interested, you can read more about our framework here.
What Was Migrated
- Migrated all 9 core METL ingestion and reverse-ETL pipelines to the new platform, covering Salesforce load and unload, Hive, BigQuery, Google Sheets, NetSuite, MongoDB, and ServiceNow.
- Migrated 664 individual jobs across 46 production schedules, validating each one against its legacy counterpart.
- Deprecated 60 legacy jobs that were no longer needed — a meaningful reduction in technical debt delivered as part of the migration, not left for later.
- Re-engineered micro-batch scheduling using a flex-connector approach that removed the environment’s dependency on SQS-based scheduling.
- Integrated the new platform with EC2 to support Python-based jobs that didn’t have a native equivalent.
- Implemented custom CI/CD automation purpose-built for the new platform, matching the workflow the customer’s engineers already relied on.
- Built comprehensive test coverage for every migrated job, including side-by-side comparisons between the legacy and new platform, execution error checks, and data validation across schema, row counts, and content.
- 2x MAIA Trainings to enable client’s Data Engineering & Operations team.
- Delivered job-level documentation for every migrated pipeline, giving the customer’s team a durable reference long after the project closed.
The Results
- 100% of METL workloads migrated to Maia. Every METL job identified in scope was successfully re-platformed, tested, and cut over.
- Improved micro-batch scheduling. The move away from SQS-based scheduling gave the environment a more resilient, flexible scheduling architecture.
- CI/CD parity achieved. The customer’s engineering team retained the automated deployment workflow they relied on pre-migration, rebuilt natively on the new platform.
- The customer’s team is fully operational on the new platform. Development and Data Ops now run in the new Maia environment, with the legacy platform slated for decommissioning.
Summary
Legacy ETL platforms rarely fail all at once — they accumulate friction: manual environment management, brittle scheduling, CI/CD gaps, and a growing pile of jobs no one wants to touch. This engagement shows what’s possible when that friction is addressed head-on: 664 jobs, 46 schedules, and 8 core pipelines migrated in 17 weeks, with the receiving team fully self-sufficient from day one and measurable technical debt removed along the way.
If your organization is still running Matillion ETL and weighing a move to Matillion’s AI-native platform, Maia, CloudEQS has the hands-on migration experience to help you get there — on schedule, with zero surprises.
Reach out to our team to talk through what a migration could look like for your environment.
Comments are closed