Facebook System Design Interview: Design an Analytics Platform (Metrics & Logging)

The design is a little superficial . In the context of monitoring systems, the crucial 'dive deep' question pertains to data aggregation and the trade-offs between storage capacity and performance.

The real world monitor system like cloudwatch and prometheus (push vs pull) have be mentioned during interview as well.

does meta ask system design question to sde1 role

Event Sourcing and projections for visualization would have been amazing here

NoSQL DB for a time series data. What a Joke!!! Can't believe FB EM giving this sort of design

Feedback:
1. should have talked more details about data storage and how the storage would support faster queries. Some sample queries as example must be shown and these queries are served.
2. No mention of how logs are stored and indexed for faster search.
3. Didn't justify the usage of queue?

Question to people proficient in designing backend systems - is this a good example of an interview or the design? I personally found this to be crossing out checkboxes in an interview. There isn't enough trade off discussions or building towards a solution. This seems like an inconsistent brain dump of a known solution.

Great video. Very natural and realistic, not like the rehearsed and phony ones like many other videos on YT.

Not very successful interview

Ok good interview with Imran Hashmi :p

This could be a case for Kafka for message processing queue with event driven API in mind..

I have a suggestion for all videos under this platform
First they are super helpful when it comes how to carry out the whole design..like how to estimate or begin with
But all of them lack cross questions from interviewee side..I mean in real life we can be bombarded with minute detail level questions
Like in this video...how is data enrichment working or how come we are making data collection to be configurable without having whole business use cases

Bottomline : make it more tougher :)

Can a load balancer directly insert to a queue?

This interview is too short so there is not enough time to talk about some of the details. The design interview should be at least 40 minutes. The candidate only had 21 minutes. There's not enough time to do deep dive and it seemed rushed. There is not time to talk about scaling the individual components. Sampling is not scaling. The interview should be longer so that the candidate can talk about how many servers are needed, how much disk space required for X number of years/month, how many requests can be served per second, etc.

The requirements list is too short. I feel we didn't spend enough time on the requirements.

How would you determine the level of the candidate based on this interview performance?

How is this guy a manager. Probably will fail intern interviews.

I think this interview will not fly. lots of flaws

Why was visualization a big piece of the discussion. Design was metrics and logging, which lacked depth. It's whole blob of logging data coming, could be stored in timeseries DB or even object store like S3 then moved to DW like Redshift. Why NOSql DB needed in this case.

This interview gonna fail, bad example

dear team, please provide the name of the tool that the user is using for drawing the architecture

Not satisfied with the discussion. Scope of the question is not clear. Are we building a system for analytics computed out of logging data or are we building a system that has logging and analytics as separate components?

Interviewee could have discussed about:

1. Grain of data that is sent by logging system. Is it individual events or aggregated counts?
2. Database design that is optimized for analyzing time-series data
3. Could have expanded machine-generated events and user-generated events and have different treatments on those datasets down the line.

Great video - Kudos to the interviewer for making the environment so comfortable. Few things:
1. I could not understand the choice of data base. Ideally this should be a combination of TimeSeries DB + Data warehouse?
2. Certain key components/Aspects like rule engine ( for acting on events), Notification systems ( for notifying the interested subscribers) were missing?
3. 90 days retention is a very less SLA. Down-sampling the data for lowering the volume and storing it for long term could have been discussed.
4. I thought that the interviewer wanted to go beyond just visualization - To automated actions ( alarms etc.) and analytics too.

Introducing a Data Catalog would really help with managing PII and auditing where and how sensitive data is being used through data lineage.

Hozefa is a beast!!!!

Heard the interviewer made mention as part of the requirements that system could scale up to a billion users meaning events could be at least double of that (depending on the metrics that want to be tracked). I think in that case maybe a nosql wouldn't be the best persistent data storage decision. Maybe an OLAP kind of database (like clickhouse) should be used. This will definitely have a drastic positive impact in the query time for both visualization (retrieving) and inserting of events and will also help in creating way faster aggregates.

Also another improvement that can be made with the design is maybe the queue can come after the validation/scrubbing service and not before. It could help save some space in the queues and not have them overwhelmed because only validated data will get into the queue and invalidated ones are discarded . Only when I see the queue should come before is if we are validating a very large batch of payload at a time then maybe we can stick with this design because validation might take some time for extremely large batches.

Metrics and logging sound like two “separate” design questions 🤷🏻‍♂️

for the system needs to be in near real time, this is a clarify question to ask the interviewer instead of assuming.

Load balancer is redundant if you're using a queue. Events should be published to the queue right away and available consumers (validation service) will handle events as they become available.

Who is the interviewer pls include her LinkedIn ID

may I know the tool name they used?

lmfao, if this guy did a real interview, he'd fail within the first 5 min, cannot explain himself, repeatedly says 'this is enough, that is enough' without actually waiting for the interviewer to confirm, and his design is shit. Talk about the scale of the system first, what kind of storage do you need? what's the throughput? who is producing the data? Who on earth gave this idiot the job of an engineering manager is beyond me. What a fucking dud

Nice:)

Not in depth at all

uhm...uhm...uhm...uhm...uhm...
....................................................is it how it works in real system???????

very good... both interviewer and interviewee did excellent job... lot to learn from this video... i have an interview tomorrow with amazon, hope this helps....