Kafka Deep Dive w/ a Ex-Meta Staff Engineer

I think your broker/partition diagram is a little confusing. Kafka defaults to 3 brokers, because to achieve a replication factor 3, you need 3 brokers with one partition (leader + 2 followers) each on each broker. You talk about a "partition going down", but it's really the broker that could go down. So if broker 1 containing the partition 1 leader goes down, then it's broker 2 (follower 1) or broker 3 (follower 2) that will take over as the leader.

This is an amazing overview holy...

How consumer will know, what is the first offset value he need to pass while reading from Kafka?

Awesome video

Thank you for sharing ❤

Need a video on comparing streams/MQ services.

Like what are the driving decisions between using Kinesis, Kafka, SNS/SQS, Redis pubsub, RabbitMQ, etc

Got a question about compound key. Some doc says the maximum partitions for a broker is 4000, and maximum 200,000 for a cluster. That doesn't seem to be enough if you set the key to AdId:userID. The amount of key will easily exceed this limit.

Great lesson!

I have a CRM system that receives events from the client (example: a card was moved, or a card was updated)

And whenever the event goes to the queue, I would like to:

- send to a webhook
- send to a socket service
- send to an automatic actions service

every event should be processed in order (order matters) and also has to guarantee that the three queues events was processed

in this case, should I use consumer groups?

Also, is there a way I have multiple consumers for each consumer group?

ty

Amazing content. I had a few questions

1. Where does the broker keep the append only log file? Is it kept on disk or in memory ?

2. When the consumer tries to read the next message, does the broker do a sequential disk read (assuming it is on disk)?

Thanks!

very well explained!

With idempotence enabled, acks=all is automatically applied, requiring acknowledgments from all in-sync replicas.

Is this bluescape where you are designing?

Is Zookeeper part of the Interview when discussing Kafka? I know Kafka can run without ZK these days but just wondering how much Zookeeper knowledge is required?

Thanks. So what are the other 4 technologies in your top 5?

This is super helpful. Thanks a million.
In the meantime, can you also shed light on the delivery mode of Kafka? In particular, how is the "exactly once" delivery achieved?

1. do consumers commit offset for every message they process or they can keep processing messages and commit offset once in a while.
2. would love to know about kafka streams and kafka channels with motivating examples :)
thanks @hellointerview & @evan!

Dumb question, but does Broker 2 need to have a leader for Topic A? Or does the Apache ZooKeeper (or some broker manager) manage partition follower replication between brokers?

This is by far the best channel for discussing system design topics. My goodness, everything is structured so well. Well done!

I feel like half way you missed the topic a bit since Kafka is not really a message queue.

absolute best content. keep up with great work :)

!!!AWESOME!!! & THANK YOU

Question: How does Kafka ensure or maintain consistency between partitions (leader and followers)?

A Dead Letter Queue in Kafka is one or more Kafka topics that receive and store messages that could not be processed

aim for 1 MB per message. And broker can take 1TB data and 10k messages per second. That would leave us with 100MB msg size which may not be a good practice. Instead , why not have 1M messages per second (given each msg of 1MB) ? cant we scale better this way ? in fact if msg size is small, we can bump up rate eg: 1kB msg (in case of some small text msgs like feeds or whatsapp) and 10^9 msgs per second.

Please do one for Cassandra as well

This is excellent

This is an amazing explanation for a beginner like me in system design! I really appreciate the videos—please keep them coming. I’m hoping to crack into FAANG companies in the next few months.

Best video I have seen on the topic. Thank you!

Thank you for the this, really enlighten me after using kafka in work for so long

Have you seen use cases with retention periods set to infinite and what would be some examples of that? retention ms and retention bytes both set to -1

Thanks for this insightful session; it is incredibly informative, well structured and highly useful for a novice or an expert of Kafka. Your method of setting real life examples and ability to emulate consumer (not Kafka consumer 😊) mindset and going behind the reasoning is exemplary; great job 👍🏼

This is such an awesome video. It was a good introduction and prompted me to then follow up on various details like how can i increase the paritition count, what are antipatterns, dynamically adding topics, adding brokers etc and other operational management considerations. This is just awesome and great job.

For consumer retries, why use a Retry topic? why not put the message back into the main topic? Is the suggestion that when putting on the retry topic, maybe we also have a "number of retries" metadata which we keep incrementing (and the original producer doesnt need this). Or is there another reason? Say because maybe we want a separate consumer group to process retries for whatever reason?

Quick question: why use composite keys? Isn't it better to then just use option 1 (no keys)?
After all, you lose ordering anyway.

Awesome video!
Quick feedback: please zoom a bit more when there's text, it's a little hard to read 😅
The font itself is also a bit hard to read.

wish i had discovered this video earlier

One of the best videos and explanation on the entire internet. Thanks a lot...

Exceptionally insightful video. Nice dive-deep

Great video on Kafka.

Question: It seems Kafka can do it all now. When do we still want to use Amazon SQS or RabbitMQ (use case) ?

Wrt. hot partitions, wouldn't it help to scale the consumer group? Or did I miss a constraint that wouldn't make this feasible?

A great video, thank you!

One thing that I would like to highlight is that the partitioning of the queue and the consumer groups are motivated by scaling, but topics is motivated by abstraction.

Ethan did mention this, but I think it was worth highlighting.

Haven't seen videos that explain things so clearly since Ali Ghodsi Data sciences lectures

This is an amazing video!! What app are you using to document the details on the video?

This is pure gold right here, my God! 🤯. Thanks for taking the time to break down these concepts.

This is amazing content. The worth of these videos is in gold. Please keep these free always. Please dont ever pay-wall these videos. Thanks again. 🙏