The Harsh Reality of Being a Data Analyst

As a data scientist I agree with all of these points…I see the same where I work

I like your point of view. I would add that the barrier of entry is low, but to be a good data analyst, that is hard. People think it's about knowing how to use the tools, but in reality it's that you use those tools to get the golden nugget, in other words, to uncover the insights and trends that drive a business. Not many people are good at it because they lack the statistical mindset and data visualization, which is what uncovers the golden nuggets. I guess when you master that, you can easily move to data science, or be an extraordinary data analyst.

I work as a financial analyst and I spend 85% of my day validating and cleaning the information I use in my tables before I run my reports. The data feeds we get can be truncated and then the data is incomplete. I work at a pharmacy and if the inventory team adds a new drug with the wrong unit of measure like ml instead of ML, the price won’t be updated on the feed. One is milliliter and one is Megaliter and the feed is case sensitive. The data warehouse is only updated every 6 weeks so it’s not the most reliable information. Some data analysts create reports with formulas that use a certain cell range but the data goes past the cell range so you need to go to the source. Processors may enter duplicate information, or not enter the information they were supposed to, or put information in the wrong fields, or make typos. We use Oracle but we get an invoice file from a supplier who uses Windows so we get conversion errors. Our devs have created code to correct the errors including what it believes are duplicates but unfortunately our supplier will break up an order of 50,000 units or more into 3 line items of 2 lines with 20,000 each and a third with the rest. Well, when the code sees the same order number and same RX number on more than 2 lines it deletes the additional lines. I have to create a ticket to recover the missing data in case there’s an audit, but my day is working backwards through the numbers and datasets so I can fix problems in external systems as well as internal systems. I can pinpoint when and where issues began, I can provide the affected invoices that need to be reprocessed to the correct price, and I do all of it without coding.

You answered your own question about pay--easier barrier to entry, lots of people can do it, you can self-teach, etc. Those are all elements of low pay.

It can only get demotivating when a lot of Data Analysts abandon their profession just because it is job that is full of frustration or a job where they feel inadequate and so forth. I hear the same thing about Data Scientist. In fact, you were the one who talked about it. Data Scienctist and Data Analyst are two different role where the former is LOT more stressfull (it is all about making profits to the companies and building creative models from nothing. It requires your very sciency abilities (total understanding of algebra and statistiics like you understand Harry Potter).
It is altogether a different story when people leave Data Analysis to pursue a different role. That does not reflect bad on the profession itself. I am still very hopeful for my future as a Data Analyst. This is what I want to be. I don't want to be a data scientist. This is the skill that will stay with me forever. I can do my own research. i write my own paper on the questions that I had about this world by backing it from the data.

It is not difficult to learn these tools but what is difficult is to learn how to analyze data and make data-driven suggestion. This is the main part of data analysis and you can't learn that watching any tutorial. This knowledge comes with experience.

Why would they have to know Python? I've never heard this before.

Data structures and algorithm use in the data analysis job??

Don't want to consider it a tech job. Then I'll analyze the data after you create the python or sql to extract it out. I'll tell you the parameters, or I'd write code at home and when I leave all the code goes with me.

Took me a year to land a data analyst job in general and that was 5 months after having a degree, so it took me at least 5 months to get an entry level data analyst job.

Thank you for demotivating.

Useful for general business knowledge, limited opportunities for a career.

Story.....

There's a glamorous side??

So What is the difference between Data Scientist , Data Engineer and Data analyst?

Thank you for the summary of career tracks and data analyst. Also you have very sexy shoulders

To your first point, granted, many Data Analysts barely analyze things and do pivot tables and such. It's good enough at some companies and brings down average salaries on glassdoor

Perfect caption and I'm glad I watched this video

Well, data analyst can be foot in the door.

I would also say anyone and everyone can become a nuclear physicist IF THEY WANT TO.

Make the L

SQL is not an easy language.

In the company I work for, I had the opportunity to cross path with data analysts. I'm a data engineer, and what I can say about your question of "How is it not a tech role?" is that being a data engineer the focus of our job are the technical parts. Data analysts deal a LOT with clients in their daily basis, it's closer to business than tech. Writing in SQL, using pandas, it doesn't even scratch the surface of how hard it is to deal with infrastructure.

Is it just me, or does she look really really tired, exhausted, burnt out? She looks like she needs sleep.

The level to entry is very difficult. Thankfully I leveraged getting my foot in the door into a tech company and was able to transfer into an Analyst role. That being said, the job market and search is flooded with people trying land Analyst roles that I’ve see jobs low balling salaries because they know someone will take that pay to just gain experience. Hopefully this changes and I still think Analyst roles are great but just know it’s not something you can easily get into. You’re going to have to stand out in some capacity.

hey Sundas thanks for making this video 👍

top of the top is an expression.

It’s so interesting finding out the rivalries and pecking orders of different niche communities. I came to data from the business side and I still laugh at the idea that within the tech bubble data scientists are perceived as more important/useful/prestigious than data analysts. In my lived experience, the opposite is true. If you’re a data analyst, maybe consider leaving the tech bubble. You’ll be able to get a lot of interesting stuff done and as a bonus, any data scientists you run into will be really nice to you because you’ll have been embraced by the business and meanwhile it’s likely no one has talked to them or used their obtuse model since messing with it once the day it went live months ago.

I can’t criticize too much, I am an engineer (chemical/pharma) by trade, I spent a lot of time behind a computer my first 2 years out of college, plugging stats into an excel spreadsheet, however much of it I couldn’t have accomplished without the analyst that I worked beside, the guy mentored me ALOT about the realities of industry. He only had a 2 yr degree from a community college but was making more than I was as a new engineer, but he was worth it, he had a ton of knowledge & certifications even some IT certs that are outside of the normal scope of an analyst but made him extremely valuable in other ways, he knew all the pertinent programming languages like the back of his hand as well. So I can’t down on the field, I know first hand it’s no walk in the park but if this guy could earn a 103k salary with just a 2 yr degree to my 70k as a jr engineer, I think something about his job was worth it for the company. Hell, I’m sure he’s probably doing better than I am now, this was 17 years ago so I’m sure he’s getting paid as he was finishing up his degree in computer science by the time I left that company. I always like talking about this gentleman when I can, because he is certainly someone that any of these analysts, data scientists, cpu scientists etc…Who are discontent in their careers can become. I personally believe it was his willingness to broaden his education, especially if you still want to remain in tech or “big data” but you want to do something else.

A harsh reality for me as a data analyst, 20 year veteran, is the fact that businesses often have non-technical managers who manage data departments. I've had some seriously stupid managers who have had a few spreadsheet skills and they think they can manage an entire data team. I have seen literally hundreds of thousands of dollars wasted due to these bozo managers who primarily exist because of super size egos and bluffing their way to the top. Case in point, I was ready to build an automation for a billing department and the COO killed the project because he couldn't understand it. The automation would have saved at least $50,000 per year. Welcome to Corporate America!

Data analysts are not nearly as selective as engineers, data scientist and especially research scientists. It's understandable companies don't see it as tech roles. Many people knows Python and SQL, but most people don't know C, C++ and whatnot like real engineers do. Most people also don't know the math behind data science, and that's part of a professional data scientist's training. As for research scientists, it requires the most training. Most of them have a PhD and only occassionally a master plus outstanding publications in a very specific field also qualifies. It takes years of solid academic training (often over 10 years) to create a research scientist.

My experience was very different. Probably because I had degrees in Math, Statistics, and Information Management. So my job was to figure out how to distill available data into useable information, and most of my work was done before any code was written. It was my knowledge of what types of analyses were appropriate that was more valuable than my ability to write code or build spreadsheets. I certainly did not learn that in a bootcamp or from self-study. I could write code too, but that's not why I made the big bucks and reported to senior management. Luckily I retired before the advent of AI, but had I not, I'd probably be developing AI algorithms.

It depends on the data maturity of the team/company. A lot of data scientists struggle to show business value whereas reports, dashboards and decks are always going to be needed. Data engineering is the most critical but doesn't really involve interacting with the business which is less interesting for some people.

I have cmplted data analyst course ,m lookng for remote job,wat things i have to consider for remote job as a fresher???

Oh I know I know! Its that it sucks, right?

She mentioned its not a tech role...but I think it should be noted as not an engineering role...the competing jobs she mentioned are engineering roles.

Roles, Titles and Pay OH MY. In big companies these things are highly defined and adhered to. Smaller companies, titles don't matter as much and pay can be all over the place.

I've found that any "Analyst" roles are never considered part of the tech tree. I've worked as a Business Analyst assigned to an IT team and while I never felt excluded, my pay was not reflective of my contributions AND my pay didn't get better until I switched to a Proj. Manager title. I wasn't doing PM work but that title got me a huge raise and it got even more when I got the title Sr. Software Developer. I was still doing Business Analyst work and even App Admin type work but they had to give me that title to get my pay level up where it needed to be. It seems silly to me but that's the way larger companies work.

What really stinks for new hires is these pay scales and limitations are often not visible. Hell even employees often aren't able to see these and have to kind of figure them out. I think this obscurity is on purpose as they can demand more from lower paying employees while having a convenient excuse to keep their pay capped.

Having analysts know and write python code, SQL code and more is ridiculous and should not be tolerated. Analysts are "the voice of the customer" and "advocates for the customer" and things like that. They're tasked with bridging the gap between IT and non-techies and while those coding skills can certainly help, it's ridiculous that companies are making those part of the requirements. Those coding skills rest specifically in the engineering or developer roles.

What's happening here is very similar to when a highly competent and highly productive employee leaves and the business thinks they can just divvy up the work to others. MANY times when a Sr. person leaves, they ask a more Jr. employee to take over until they can find a replacement. Often this replacement never shows up and after the Jr. person learns the work, the business finds they get the same things for less money. When the Jr. person asks for a raise and new title, they're often shunned, scoffed at and manipulated into still doing the same work for the same shit pay because "Well we're in a hiring freeze right now" or some other BS excuse.

Companies are masters at playing this game and employees would do well to not stand for such practices. THE MOST egregious examples of this are when non-managers are shoved into doing managerial roles and never given the title or raises for the additional work. The same thing has happened with Data Analysts only it's happened so much faster people don't realize it and simply put up with having to know SQL, python and more as a way to land these lower level jobs.

Don't put up with it.

If you have to know Python and SQL for a DA role, it's 100% NOT a DA role....it's a developer, engineer and/or other highly tech role and should be compensated appropriately.

.. where was the 'harsh' part .. ??

And why data analysis can’t be done by software or AI?

What are your thoughts on the CAIP designation?

You confuse people too much! A person who is looking for guidance, instead of helping, you play your content-consuming game!

Go for it and "get great deep with Python" and other tools, there is 'tech jobs out there" and "solid skills lead places"

Solid Skills will squash job interviews and lead to so many Python deeper jobs!

Is anybody else having a hard time finding jobs? I graduated last december with an applied math and stats degree from berkeley and havent heard back from any applications. I can do neural networks, deep learning models, machine learning, forecasting, programming, etc. Its been so hard finding jobs and I dont know why

Thanks, Love You.

SQL is not a coding language