Matillion Has Got Some SaaS

Overview

In this article, we’ll provide our perspective on our key partner, Matillion, and their evolution to becoming a SaaS company. As the world of data has evolved from analytics to the AI-era, so has Matillion.  

Who is Matillion  

Matillion is a data integration, transformation & orchestration provider. They first came to market in 2015 with their flagship product Matillion ETL (METL). The intuitive interface coupled with robust functionality to ingest data using pre-built connectors, transform data, and orchestrate pipelines meant Data Analysts, Engineers, & Architects could scale out their data pipelines. Over the years, adoption of their platform was adopted by leading enterprises such as Cisco Systems, Autodesk, Slack, Western Union and Yext. 

From a deployment and architecture perspective, METL was deployed via the AWS/Azure/GCP Marketplaces where customers would install a VM within their cloud environment. At the time, this innovation was huge and allowed customers to securely download the software, adhere to their organization’s security/GDPR/VPC requirements and get up and running in a matter of minutes.  

Why Move to SaaS 

As a systems integrator and Gold Partner of Matillion, we’ve worked with clients to optimize their METL platform. Based on customer feedback, there are some reasons why Matillion has built a new SaaS platform, The Data Productivity Cloud: 

1. Reduce infrastructure cost 

2. Reduce management cost  

3. Increase scalability 

4. On-board more teams 

5. AI integrations 

Let’s look at a couple examples. In the METL world, customers had to manage the VM & software updates. This meant clients needed to stay on top of release cycles – which happened every 6-8 weeks – and embed them into their project plans. Oftentimes resulting in slower deployment efforts, reliance on additional teams to get new VMs provisioned, and, ultimately, inhibiting the ability to onboard new divsions.  

Another example is scalability. Since throughput was tied to the number of cores of the underlying machine, customers often have a stair-stepper approach to scaling their workloads. Additionally, compute resources on the underlying VM were intensive and could result in hanging UIs and high CPU utilization. Resulting in spending additional time trouble shooting and fixing as opposed to building new pipelines that delivered value to the business. 

Time to Get SaaSy – The Data Productivity Cloud 

Matillion’s Data Productivity Cloud (DPC) is their brazen declaration that they’re entering a new chapter. The facelift, changes to the underlying compute engine, re-architecting their GIT integration, and AI-capabilities means data teams can be widely more productive.  

To start, let’s talk facelifts. Matillion’s Data Productivity Cloud still has the same user experience but with a more modern feel.  

Now Let’s Talk Agents & Planes 

True to their UK roots, Matillion has channeled their MI6 energy and deployed DPC using stateless containerized agents.  Additionally, they’ve separated the control plane from the data plane. Meaning, all software updates are automatically handled and pushed to the control plane. As a result, customers no longer need to manage upgrades and Matillion pushes 2-3 releases per week! 

From a deployment option, customers can select between full SaaS or hybrid SaaS depending on their requirements. See full documentation here.

GIT Back To Where You Once Belonged

Next, GIT is at the heart of the Data Productivity Cloud. Now, users can seamlessly perform GIT commands (commit, pull, push, & merges) from within the UI. Resulting in increased collaboration across teams, zero UI lags, and more control over code changes.  

AI Boogie Wonderland

Saving the best for last, the AI integrations & innovations. Matillion has released a breadth of AI components to accelerate development, extend into RAG use-cases, and unlock AI workloads. Including:  

• AI Notes: invoke generative AI to annotate your data pipelines 

• AI Prompts (OpenAI, Amazon Bedrock, Azure OpenAI): take inputs from a source table, combine the inputs with user prompts, and send this data to the LLM for processing. 

• Pinecone Vector Query: ingests text search strings and returns text data associated with similar vectors in your Pinecone vector database. 

• Pinecone Vector Upsert: converts data stored in your cloud data warehouse into embeddings, and then stores these embeddings as vectors in your Pinecone vector database. 

• Snowflake Cortex: Integrate with Snowflake Cortex’s functionality – Translate, Summarize, Extract Answers, Completions, Sentiment, Embed, Vector Search, and Document AI Predict  

• Snowflake Vector Upsert: convert data stored in a Snowflake cloud data warehouse into vector embeddings, and then store these embeddings in a new Snowflake table 

Summary 

In summary, Matillion’s Data Productivity Cloud is the direct answer to the headaches their Matillion ETL customers were experiencing. Customers aren’t inhibited by the size of their machine thanks to DPC’s stateless agent technology, upgrades are automatically handled, GIT is at the heart of the product, and AI integrations open the breadth of use-cases for unstructured datasets leveraging LLMs.  

Safe to say, we’re excited about the future of Matillion and helping our joint clients get their data AI-ready, faster!