What Is DBT and Why Is It So Popular - Intro To Data Infrastructure Part 3

OMG Ben, I also learned from Wise Owl Tutorials!

feedback for your blog - the text colour lacks contrast to the background - makes it very uncomfortable to read

Still don't get what is dbt

@SeattleDataGuy , Thanks for your contribution in help many understand DBT concepts.
I have implemented an incremental DBT load(merge) with ProstgreSQL, and the target table is partitioned.

The load is performing too good in lower environments and taking half the time it used to take before partitioning.
Strangely, when i move the changes to production, its taking double the time it used to before partitioning.

Same data volume,
Similar DB configuration
only difference is, lower env, its server less and prod postgresql is not serverless.

any thoughts around this would be greatly appreciated.

You want to talk about dbt vs dagster? Airflow is kinda nuts using for ETL imo. Airflow isn’t designed for ETL at all - dagster is.

dude how r you, i have the next question, what could i do if i have a stream on snowflake that i want to "consume" in dbt but not creating a physical table or view, instead something live a ephemeral materialization, only to purge the stream and avoid to become stale. I create an ephemeral model and select the stream source but that only create obviously an ephemeral materialization but kind not clean the data on the stream, thoughts??

Is this yet another "platform independence" obsession driven solution?
Why should a company, that has decided to go with the Microsoft stack or the Oracle stack or whatever, reject the possibility to use its product specific tools, that give you all the freedom to create your custom ETL processes, while profiting from all specific features of your stack??

Thanks! I have been sleeping under a rock for two years - appreciate you helping me get up to speed 🙂

After watching this video, it is still not clear to me what exactly dbt is doing. Examples would have been great.

Thanks for the explanation!

SQL and JINJA! 🤯 My Mind is blown!

I'd say SQL is a Pro. If you've worked in BI for any length of time then you know some SQL, so adopted DBT will be very smooth.

Sorry, but I don’t search for technical topics to see a talking head.

Greetings from Vancouver. Thank you for this video on DBT. All other ones were 30min+ :)

How would dbt replace adf when there to E AND L in dbt? Who’s gonna extract and load data into the warehouse for dbt to be able to transform it?

DBT was a game change for our tableau workbooks. Our tableau workbooks were taking sometimes up to 1-2 minutes to fully load. Once our data engineers added DBT to the mix, the tableau workbooks were taking 3-5 seconds to load. I am not fully sure how all this was done by our data engineers but it was straight up voodoo magic and our end users loved it. Bonus is it made us look good to our 3rd partners who said even their own data team's reports were taking minutes to fully load.

Okay i downloaded dbt why is my code still awful

very high quality content. Helped me understand some long running doubts I had. Thankyou for sharing

Meltano... Such a disaster... Loved by millions....right..

super helpful content, thank you!

Thanks 🦌.

watches first 40 seconds So let me get this straight: you think most people know what DBT is, but it has only recently been understood that VCs can overvalue startups?

Nicely Explained - thanks

I can correlate with your journey. Started off with using DTS in SQL 2000 and using SSIS first time with SQL server 2005 :)

I always struggle remembering that DBT is not only acronym for Dialectical Behavior Therapy

Hi Ben! Your videos are always top-notch and very helpful! I'm wondering what "snp" (?) stands for in this case. dbt would be an awesome tool to learn!

Dbt la jazates ka nahi kulthe

So it is because its' a watered down ansible to the point someone would only need to know sql?

Wholly goodness thank you. I've been learning dbt as a Software Engineer and have been trying to figure out why it matters at all. This video helped a ton!

Basically Terraform for Data folk.

Loving your channel and this video. I had a similar start with data pipelines and to me dbt feels like somebody took the frustrations I had with those GUI ETL tools and packaged it up into a better, modern product. Writing code is more fun anyway (IMO). Keep up the great vids!

I’m a new Data Engineer and I’m scared to ask what is “Jinja” 😅

thanks for the video ben!

dbt was a breath of fresh air from ssis. I still think ssis was great for its time and still a usable tool in some places. sometimes I miss it... sometimes.

Great Video man!

Hi Ben, do you think no code etl tools like Informatica cloud are way more popular because it's easier to find resources? Or is it just because iics is very mature in terms of implementation.

We use airflow + DBT in most of our projects now, it's such a good tool

You always know what I want to watch. LOL
startdataengineering is 🔥🔥

My question is at this point are there TOO Many tools?

What do you see in the market; companies using python frameworks (SQLAlchemy, airflow etc) or no code tools like DBT(not considering SQL) more?

Edit: i meant SSIS, informática not DBT

dbt is a great tool. I would really love to see a video from you comparing Prefect, Dagster, and Airflow and how they fit into modern data stack.

It's about time! Excited to watch this.

Elite thumbnail

literally just had an interview yesterday where they asked me if i have experience with dbt and i said idk what that is lol

Great video as always Ben! Thank you for consistently providing us with value.