DBT (getdbt.com) on Exasol

[torsten]

Hi everyone,

in case interested there is a dbt-exasol adapter that I developed with my colleagues. It currently works, but needs a little patch for dbt because of keyword usage. I am currently trying to get the patch into the dbt main release.

--> getdbt.com

--> github.com/tglunde/dbt-exasol

In case of any questions or other bugs/problems please feel free to contact me. 

Regards, Torsten

[exa-Chris]

Thanks Torsten for sharing this, lets have a quick call and discuss how we can further promote and feature this in the community. Please DM me in the Community

Regards

exa-Chris

[jwarlander]

One of my colleages tipped me off about dbt, it seems like a great tool and I look forward to playing around with it!

Torsten, what has your experience been so far? Any particular things to keep in mind for Exasol specifically?

[mucio]

Hi Torsten,

I have some free time to check out the Exasol/dbt integration, what is the best way to contact you?

Thanks,
Mucio

[torsten]

Hi, we could have a zoom? Please send me an invite to torsten.glunde@alligator-company.com. Regards, Torsten

[torsten]

Well, experience has been good - great in fact. Since it is all about the SQL being prepared and sent directly to Exasol there is really not much that could be wrong in regards to DBT and Exasol - because the added layer by DBT is very lightweight and flexible. You decide on your own and depending on the project how much DBT/Macro/Jinja additions are to be introduced in your project or as needed. 

Most important is do some quick tests in the beginning with your sample data and tooling - then take a step back and have some team meeting to agree on project standards regarding SQL-standards, project layout and data architecture. But the DBT defaults from their website are quite descend already - not much difference there between Exasol and the other DWH / Analytics databases (except Exasol is the easiest and fastest, of course :-))

Regards, Torsten

[SusanneK]

Hi @torsten , I am really curious about dbt, but  have read several descriptions and even watched videos an I just cannot get the point. What is the advantage with dbt exactly? I think in Exasol we can do all of this with scripts as well. What am I missing? I would really appreciate your opinion on why dbt would be a great tool for Exasol users.

Thanks and kind regards

Susanne

[torsten]

Hi @SusanneK ,

technically you can do everything in scripts in Exasol. But the advantages of using DBT are

- modularisation of your transformation code, integrated dependency control for your execution

- documentation and lineage graph for those transformation - generating useful catalog and end-user documentation

- integrated testing - standard tests on columns for uniqueness, not-null, relationship, expected-values - and sql based tests

- TDD with seed data - to run your pipelines in CI with subset of data

- combining code (SQL+Jinja templates) and documentation (YAML files + Markdown) you have one combined codebase to release everything together

- re-usage of code through ready to use packages - hub.getdbt.com 

- through the jinja templating you can enhance code re-usability

- Enable DataOps with CI and CD pipelines (through execution on DAG with tags)

- if you want to pay - cloud.getdbt.com for execution

Cheers, Torsten

[SusanneK]

Hi @torsten, thanks a lot for this detailed answer 🙂

kind regards,

Susanne