dbt integration with Holistics
Holistics fully integrates with your dbt project, allows you to perform data modeling and transformation at dbt layer, and push those definitions to Holistics BI layer.
(This feature requires your Holistics account to be on version 4.0)
How does Holistics work with dbt?
If you’re using dbt and a BI tool, you may face these problems:
- Disconnected data refreshes: when your data in underlying dbt tables updated, your BI reports might still be using the cached, stale version.
- Duplicated metadata definitions: All your fields’ descriptions defined in dbt are not being exposed to business users accessing the BI interface.
- Incomplete Version Control: Your data logic is version-controlled in dbt, but not in the BI tool. Wouldn’t it be nice for all your BI reports to be version-controlled as well?
Holistics BI fixes these problems by integrating deeply with dbt, allowing data and metadata to flow between the 2 systems seamlessly, and also allow your BI code to be version-controlled in the same code repository with your dbt code.
That way, you can leverage dbt as a central place for data documentation, what can be synced with Holistics, and use Holistics as the the central entrypoint for data consumers.
1. Transform data in dbt
Use dbt to transform and materialize into data tables inside data warehouse.
2. dbt models mirrored in Holistics
dbt models and data are loaded into Holistics for further manipulation.
3. Define metrics in Holistics
Define custom measures and formulas in Holistics for exploration.
4. Build reports and datasets for end consumers
End users perform self-service exploration based on predefined datasets and reports.
How does the final setup look like?
You use dbt for the transformation of data into pre-aggregated tables. You model those tables (i.e materialized views) into Holistics. In Holistics, you define logical metrics and formulas, and turn those into datasets and reports for end consumers.
When your dbt models’ data are refreshed, Holistics automatically triggers refreshes to your BI reports. Changes to your dbt medata are also pushed over to Holistics.
Both your dbt logic and BI logic are stored in the same Git repository, making it easy to manage and maintain.
Auto sync
When you change metadata in dbt model, Holistics automatically picks those up and reflect into the BI layer. Relevant datasets and reports will be updated.
Exposing dbt metadata to business users
Business users can get access to schema metadata that data teams define in dbt docs.
Continuous flow trigger
When dbt runs and underlying table data is updated, trigger will inform Holistics. Holistics can refresh data in relevant reports using that model.
Single code repository
You can maintain a single Github repository with both dbt and Holistics (AML) code. This represents the customer's analytics pipeline as code.
I'm doing all my modeling in dbt, must I switch to doing it in Holistics?
No you do not need to, and you’re not meant to. Holistics is designed to complement dbt.
dbt is great for doing most of your central definitions modeling, but you will face some constraints when it comes to modeling metrics. For example, in dbt you cannot define non-additive metrics (unique calculating metrics like DAUs/MAUs), this has to be done outside of the database.
That’s where Holistics modeling comes in, you define logical metrics in Holistics, and only resolve to SQL query when the end-users explore the data.
You use dbt as dimensional modeling layer, while Holistics as metrics modeling layer. In short: “Metrics for Holistics, Dimensions for dbt”.
Frequently Asked Questions
What are the things should be done in dbt, and what should be done in Holistics?
You can use dbt to clean, reshape and preaggregate your data, then use Holistics modeling to build custom measures and define relationships to allow Holsitics to write SQL queries gainst your data schema.
What if I don't use dbt? Can I still do dimensional modeling?