The Harsh Reality of Being a Data Engineer

Hi jash , I'm a sql developer and im going towards to being a data engineer .
My question is how you know what you should learn for now and for future ? Which community you choose as your source for being aware that what is right technologies to learn ?

I am in class 10

Jash wanted to know if an analyst with 6+ years of experience can transition into data engineer roles?

Hi sir it is really difficult to connect with you. You have not accepted my request on linkedin been pending for months i guess. If i cannot afford 1600 rupees does that mean I cannot get career guidance from you ? I guess google pays really handsome amounts still I don't know why people sellout.
Khair its your decision, Thank you mjhe career guidance chahiye to be specific if you can hear me out would be really great for me.😊

what bothers me the most is multiple tech stack doing the same thing, not getting fit because of it, and the job opportunities are still less.

As a data engineer, I can confirm these points.
Migrating to new technology involves a lot of stuff.
One of the point that I can think of is the architecture part. As a data engineer, you are often given a responsibility to make architecture that satisfies all the criteria that exist in the client ecosystem. This involves communication between several teams from network to infrastructure to security.
You can't go on creating pipelines n all before getting your architecture approved.

Now I get it. why you switched to ML.

My sister is data engineer and I am very proud of her. She works 10 Hours a day. Some times i feels sad for her for her long working hours and it is taking serious toll on her health.

Data Scientists earns more than DE.

Hi jash...which cloud will create more data engg jobs going forward?

Can you comment on project pro.. Does it really help.. Does it have real time projects which industry uses, what did u feel personally and do you recommend it

Coming from a datawarehouse background and I hear and see DE and DS I remember the need of data architecture and the use of a ETL tool for datawarehousing and BI. DS and DE give me the impression of anything goes and quick and dirty and after a while these projects tend to go to more structured, scd and maybe a need to use an ETL tool to realise that. So I miss a bit the added value of DE and DS over old school datawarehousing.

Awesome !! love you bro.... your content is really helpful.

Completely Relatable Bro....

the amount of times i had talks with C-level managers expecting something like a datalake solving aaaall their analytical problems ... completely oblivious about it's not more than a tool .. enabling fast processing of large data amounts and grab and forget data collection .. but it's not providing value .. (yet) all the work has to be done anyways ..no matter which tool is used ..

Isn't that true for the DevOps/SRE roles too? Being a DevOps Engineer myself, I always feel like a "Second class citizen" in the sense that my work has little to do with the aspects of the application that are driving the whole business and is mainly about just getting it available to the customer. Sure we're the ones to ensure the availability, redundancy etc and help the organization get rid of mundane manual stuff but i never felt that sense of longing. @Jash Whats your take on this

Hi Jash Radia, I am From Nepal 🇳🇵. A 19 y/o boy having a curiosity to learn Data Science. Can you make a video on the courses that are available online for data science step-by-step. If you can !! It means a lot to us ❤

Hello sir. Sir I m from commerce background currently I m working as MIS accounts and want to shift in data engineer line could u plz guide me?

as a fellow data engg, working in service based company, i can relate with you. stress is high and WLB is low also pay is less :( we are literally treated as second class citizens and other people will not give us the same respect like SDEs get. also learning part is so much correct. i literally spend some extra hours everyday to learn new tools and tech that i dont get time to spend on other things. starting to regret in data space. its better to be frontend or backend engg who has great pay, respect, high wlb and most imp less learning than DE.

Can i get a data engineer certification instead of a college degree.

ChatGPT will kill Data Engineering within an year

I have total 8+ of years expereince in IT and I have more working expereince on sql and working exp of 3 yrs in informatica. Whaat do you suggest for me to learn new tech is it data engineer or devops is better ? Please suggest😢

I worked as a Data Engineer for almost 3 years (Spark application development). I agree with most of the points here. Except few,
1. Data engineers usually have more work compared others like Business Analysts / Data Scientists during the application development.
2. Doing dev work in DE usually comes with some risk. Any bugs in your code has substantially bigger impact. You will be accountable for any errors in data processing. Also it is very difficult thing to track what caused the issue and since when.
3. Testing the correctness of application usually takes longer than other software systems.

Hey jash can u please guide how to land a job in google

I don't agree. Earlier I used to work in a product based company and I built a data product which is marketable to the clients.

You can use pangres.upsert() in python for update+insert based on the scenarios.

finished watching

Hi jash
Serious question
I am currently in support role
Wanted to switch my career to data engg.
I am avg in coding will it be suitable for me
Please answer as soon as possible.
I want to make this decision quickly

Can non it background get job after learn data engineer or they should go for data analyst

Please help!! very confused now a days. Im mechanical engineer having 4 years of experience. I quit my job and learn Data analytics skills like powerBI, python,sql.. now someone suggest me go for DE . Great opportunity in future. DA job are less according to him.. wht can i do.please help. Should i starts learning DE? Or focus my existing skill for DA?

U r looking like Satish ray

Thanks much Jash for the details . Iam currently working as oracle dba in rotation shifts.Could you suggest some job with less coding skills

one thing i know is that you love witcher games haha

I just couldn't help but stare at that shirt...................

From data analysis to transition which would be best data science or data engineer ?

Underrated video 💪🏻
The overview you made is clear enough. What I don't understand: is it easier to get a job as a full-stack web developer than as a Data engineer? If I understand what you say, it seems that data engineering requires more skills, but that's not explicitly said.
My idea is that employers will more easily hire a web developer than a data engineer.

All valid points. In cost centers, or CoC, where you are not under the CIO organization, your direct lead is non technical person and you are limited to tools associated with no or limited cost, then you are Data Engineer without described harsh reality, its even worst, all is hyped to Data Science, no one want to invest time to data governance.

I agree

Regarding scd type II, how do you explain them to users or business analysts. I had a lot of problems with analysts just using needless transformations to visualise such a tool, as opposed to giving them a long format table that they can visually transform easier. I know that a slider can solve the problem, but I found it that it can take more time to find the solution for them, than just give them what they want and accept my faith. Just curious

Need advice everyone
For someone who isn’t a big fan of coding, which role will be most suitable data analyst or data engineer (data scientist is coding algo heavy so not going that side)

FYI - ProjectPro is terrible. Poor Content.

thanks buddy!!

First time visiting your channel. I really got a laugh from the interview skit. 😂

I have been a data analyst for 6 years now, current in a lead role, currently earning over 25 lpa (fixed). Have super high interest in this role but changing careers is pretty daunting. How much would does a DE role get paid for someone starting afresh?

After that opening skit, I love this guy!

Nice try
Start is copy of "joma tech startup" video