Overview
Last month was DBT Lab’s annual user-conference, Coalesce, where they announced their product vision – The Control Plane for Data Collaboration. As trusted advisors and Data Engineering experts, the purpose of this article is to provide context leading up to Coalesce, synthesize the vision for DBT, share the key announcements and product features, and where we see value for our clients.
The Lead Up
Leading up to Coalesce, DBT Lab’s CEO released another one of his marquee letters to the market, the Analytical Development Lifecycle (ADLC), as a framework to mature analytical workflows. The framework centers around applying software development lifecycle principles to the data & analytic industry.
If you’re interested, you can read more about the ADLC here.
Building off the ADLC framework, Tristian announced their core ethos for how they are building for the future – One DBT. The data world is demanding more governance, standardization, and interoperability across clouds, data platforms, and skillsets. Thus, One DBT, is their philosophy that underpins their announcements.
The Control Plane
Announced under the One DBT ethos, the major announcement coming out of Coalesce is DBT moving towards being a Control Plane. What this means is centralizing metadata across the orchestration, observability, cataloging, and semantic capabilities of the DBT Cloud platform.
The product feature announcements from Coalesce were:
-
- Visual Editor – Democratize data modeling using a user-friendly, drag & drop UI for creating and editing DBT models.
-
- Cross Platform DBT Mesh – Connect and manage data across multiple platforms. Increase governed collaboration across multiple DBT projects, data platforms, and DBT deployments.
-
- DBT Copilot – an AI-powered assistant to simplify the data analytics lifecycle.
-
- Advanced Continuous Integration (CI) – Ensure higher quality of code and smoother collaboration amongst teams via automated testing, validation, and deployment processes.
-
- Support for Apache Iceberg – Leverage the benefit Iceberg’s schema evolution, hidden partitioning, and time-travel capabilities from within their dbt workflows.
-
- Auto-exposure with Tableau – the dbt DAG can be automatically populated with downstream exposures in Tableau.
-
- Microbatching – Allow incremental model authors to load and backfill large models in date-based batches.
-
- Data Health Tiles – Embed health signals within any dashboard.
Let’s double click on a few of these.
Visual Editor
There is always some level of data prep needed for analysts to translate their business context into actionable insights. Though, those with the business context typically have 2 actions they can take: (1) create a ticket for their (overworked) data team hoping they will get the right data back in a timely manner or (2) DIY with a CSV pull and a spreadsheet. We all know neither approach is scalable.
Enter the DBT Visual Experience
With the new visual experience, users are able to solve typical data-prep use-cases by translating domain knowledge into real analytics code.
As you can see, users can visualize data lineage, create and modify models without writing SQL code, and automatically generate SQL based on visual representations.
Cross Platform dbt Mesh
Cross Platform dbt Mesh is the answer to the data platform era we find ourselves in where enterprises rely on numerous data platforms across different cloud providers. With this enhanced architecture design, users will be able to reference and re-use data assets from different projects and data platforms.
Here’s an example of this architecture:
DBT Copilot
This is DBT Cloud’s embedded AI engine to accelerate & automate analytics. As of today, this includes:
-
- Auto-generate tests, documentation, and semantic models (all in beta)
-
- AI-chatbot powered by the dbt Semantic Layer (in beta as part of the dbt native app in Snowflake)
-
- Bring your own OpenAI API key (GA).
In the coming months, dbt Copilot will extend to help automate code generation and workflows across all of dbt Cloud.
Compare Changes with Advanced CI
Continuous integration has always been a strong suite of DBT and one of our standards when we implement DBT for our clients. Their new compare changes functionality doubles down on the value of reducing production bugs, improving data quality, and increasing trust in data.
When enabled, each CI job will include a breakdown of the columns and rows that are being added, modified, or removed in your underlying data platform as a result of executing your dbt job. Additionally, users can see a summary of these changes inside the actual PR in their Git provider interface.
Value
Like any software company announcing product enhancements, it’s tough to decipher which features will drive business value. From our perspective, here are the features announced at Coalesce that we’re most excited for our clients and potential ROI:
-
- The Control Plane – Increase data governance & data management by standardizing transformations on DBT.
-
- Visual Editor – Increase the number of employees leveraging data, enable data engineers to focus on data engineering tasks, not analyst tasks.
-
- Cross Platform DBT Mesh – Remove restrictions based on data platform providers and improve collaboration across divisions.
-
- DBT Copilot – Accelerate development & automate tasks for users
-
- Advanced Continuous Integration (CI) – Ensure data integrity, reduce production bugs and, ultimately, improve trust in data
-
- Auto-exposure with Tableau – accelerate root cause analysis via visualizing lineage of DBT models from within your BI platform
Conclusion
In conclusion, the momentum behind DBT Labs is strong. Customers see tremendous value in their products, their leadership has a great grasp on the market, and their new capabilities double down on solving critical problems for their customers. As the industry shifts towards more standardization (in both tools & workflows), we view DBT as a leader in data transformation and look forward to collaborating with our clients innovating on top of DBT.
Comments are closed