Data Engineering Course for Beginners

In the "Building a Data pipeline from Scratch" section, when I am running the containers by docker compose up, I get the following error:
elt-elt_script-1 | pg_dump: error: aborting because of server version mismatch
elt-elt_script-1 | pg_dump: detail: server version: 16.1 (Debian 16.1-1.pgdg120+1); pg_dump version: 15.5 (Debian 15.5-0+deb12u1)
I have installed the latest version of postgres on my machine i.e. 16.1. I have also removed the images and volumes and rerun the docker compose up command, still I get the above error. Can someone please help? Thanks!

these are all using free software right??? gcp, azure are annoyingly expensive

have a issue with postgres version mismatch. I change the version of postgres server to suit with it.
services:
source_postgres:
image: postgres:15.5

same for destination_postgres.
Hope this help

another one had issues with host.docker.internal hence added extra_hosts:
- "host.docker.internal:host-gateway" to docker compose for dbt

Im curious what tools, besides, Airbyte can be used? (the presenters employer - FYI, this is a marketing channel for Airbyte).

Came across another issue while running code for datapipeline gave me version error between dump and postgresql I changeed the version from latest to 15.5 to match dump version... hope this helps in case anyone face issues on this module

Can you make data engineering boot camp.

Thanks for the course .. to all those using VM's make sure files are located on the VM .. tried running docker excercise with docker on VM and files on host (Windows) wasted a lot of time resloving errors finally moved all the files to the Ubuntu VM where things ran smoothly ...

This is just Docker ad ? What the heck?

the best DE course that i ever seen. the most courses only stick to the theory and never show the practical part.

Can we use docker volume as a database instead of sql?

The dbt section might have been easier to explain if it was installed to the base image; all of the config could have been completed in the docker file?

As you are using VS Code, it might be a good shout to use the remote conatiner extensions to interact with the running containers, too.

Anyone facing issues with pg_dump in etl_script, using postgres image 15.5 would fix it:
You need to use version 15.5 for the postgres.
services:
source_postgres:
image: postgres:15.5
ports:
– “5433:5432”
networks:
– elt_network

Not beginner friendly, to be honest

the film sql example is terrible. Very complicated and abrupt way to explain such simple concept as INNER JOIN.

Here's your data model. People/Places/Pennies/Product involved in a Process.

for everyone in the comments asking for a "bigger course", you could always supplement this beginner course with DataZoomcamp's Data Engineering course.

bhai kya faltu padhata he

Hey , currently there is no set path to becoming a data engineer. So please create a proper certification with a clear roadmap of foundations and most used cloud tech in data engineering so that we can get some structure going for those interested in this career.

Thank you for amazing video. Could you please a second video including Data Engineering Projects

Finally a data engineering course!

SiR i need full data enginnering adVance course that can get me a job at FAANG

You know when a data engineer beginner course gets uploaded on freecodecamp, you can confirm that data engineering is actually growing and getting popular now.

best video on the internet

This is excellent content

Finalllyyyy an data project. What are the learning pre-requisites for this course?.

Data Science ❌
Data Engineering ❌
Data Analyst ✅

Yes! Thanks!

Good job, you are creating visibility for airbyte in a great way, by providing an evolutionary view of the stack that gets one to eventually need it. Hope they continue to support you making content using this approach.

Is data engineering at risk due to AI?

lol wasn’t this guy into web development 😅

Has anyone ran into trouble with this. I was following along, and I got this error
subprocess.CalledProcessError: Command '['pg_dump', '-h', 'source_postgres', '-U', 'postgres', '-d', 'source_db', '-f', 'data_dump.sql', '-w']' returned non-zero exit status 1.

No matter what I do I still get this error. I pulled his code, still didnt work. I went on another computer and tried and pulled my code and his code and still the same exact error.

oh, no. Can you add subtitle to video, please?

thank you very much

Finally!

Please complete j2ee

ChauCodes !

YES THANK YOU SO MUCH keep expanding this!!!!

Love your tutorials. But can you make a c++ sfml with visual studio tutorial that would help me a lot. Thanks

We need a 60 hrs course for DataEngineer.

ok, this is super helpful.

Real GEM

What are the learning pre-requisites for this course?

Do Justin Chau have a virtual adress?

please come up with big data course