Matillion’s AI Features

Overview 

One of our favorite things about Matillion is their agnostic approach to solving complex data problems. Customers will have a variety of data platforms and cloud providers and Matillion doesn’t shy away from meeting their customer needs.  

Sticking with this ethos, Matillion’s Engineering & Product teams have been innovating at a rapid pace. Thanks to their underlying SaaS platform, Data Productivity Cloud (DPC), they’re constantly pushing new features every week. 

Thus, the purpose of this article is to highlight their AI features, value-ads, and practical use-cases. 

Matillion AI Features 

Here’s a list of all of new Matillion’s components that leverage AI within the platform and/or unlock AI workloads:  

• • Copilot 

• • AI Notes 

• • Open AI Prompt 

• • Azure OpenAI Prompt 

• • Amazon Bedrock Prompt 

• • Pinecone Vector Query 

• • Pinecone Vector Upsert 

• • Snowflake Vector Upsert 

• • Snowflake Vector Search 

• • Amazon Textract Input 

• • Amazon Transcribe 

• • Azure Document Intelligence 

• • Azure Speech Transcribe 

• • Document AI Predict 

Value-ads 

Let’s look at some of the value these features unlock. We’ll break these into 4 sections: 

• • Developer Productivity 

• • Prompt Engineering 

• • Retrieval-Augmented Generation (RAG) 

• • Unstructured Datasets 

Productivity 

Starting with Matillion’s copilot. Built on top of their already intuitive UI, Copilot lowers the barrier of entry even further.

 

Lending itself to the citizen data developer, with Copilot, users can create data pipelines using plain language instructions. Once the statement has been made, Copilot uses the underlying knowledge of the platform to bring up the necessary components for the job.  

Secondly, is AI Notes – the Claritin for Data Engineer’s allergy to documentation. With this feature, users invoke generative AI to contextualize each component in the pipeline. Users can verify, refine, and edit the responses.  

Prompt Engineering 

Matillion’s integrations with OpenAI, Bedrock, and AzureOpenAI unlocks prompt engineering workflows with the latest LLMs. With these features, users design inputs –prompts – to guide generative AI models to produce optimal outputs.  

The component works by taking one or more inputs from a source table, combining the inputs with the user prompts and sending this to the LLM for processing. Users can configure the component to output the results as either a text value or JSON object for further processing.  

Retrieval-Augmented Generation (RAG) 

Retrieval-Augmented Generation is a natural language processing technique (NLP) to optimize the output of am LLM. It does this by referencing an authoritative knowledge base outside of its training data sources before generating responses.  

With Matillion’s Pinecone Query/Upsert and Snowflake Vector Upsert/Search components, users can bring vector embeddings to LLMs to create more accurate and relevant responses. 

Unstructured Datasets 

The proliferation of unstructured data has always been a problem for Data Engineers & Architects. But with generative AI, this is becoming a problem of the past.  

With Matillion’s integration with Snowflake’s Document AI, Amazon’s Textract & Transcribe, and Azure’s Document Intelligence & Speech Transcribe, users can now tap into the wealth of insights buried within PDFs, audio transcripts, and handwritten notes. 

Under the hood, Matillion integrates with the APIs of these services within their pipeline and converts the unstructured data into semi-structured. Once converted, users can continue to manipulate the data like any other pipeline.  

Practical Use-cases 

Here’s a short list of use-cases we’ve identified within our AI lab: 

• • Customer Chat Bot & Ticket Creation 

• • Customer Sentiment Analysis & Churn Prediction 

• • Product Roadmap Enhancements 

• • Financial & Compliance Reporting 

• • Content Creation  

Conclusion  

Matillion’s new AI features further their mission of making data practitioners more productive. It expands their platform into new workloads, allows customers to leverage cutting-edge technology, and extract insights from the plethora of unstructured datasets within an organization.