Комментарии:
I think your broker/partition diagram is a little confusing. Kafka defaults to 3 brokers, because to achieve a replication factor 3, you need 3 brokers with one partition (leader + 2 followers) each on each broker. You talk about a "partition going down", but it's really the broker that could go down. So if broker 1 containing the partition 1 leader goes down, then it's broker 2 (follower 1) or broker 3 (follower 2) that will take over as the leader.
ОтветитьThis is an amazing overview holy...
ОтветитьHow consumer will know, what is the first offset value he need to pass while reading from Kafka?
ОтветитьAwesome video
ОтветитьThank you for sharing ❤
ОтветитьNeed a video on comparing streams/MQ services.
Like what are the driving decisions between using Kinesis, Kafka, SNS/SQS, Redis pubsub, RabbitMQ, etc
Got a question about compound key. Some doc says the maximum partitions for a broker is 4000, and maximum 200,000 for a cluster. That doesn't seem to be enough if you set the key to AdId:userID. The amount of key will easily exceed this limit.
ОтветитьGreat lesson!
I have a CRM system that receives events from the client (example: a card was moved, or a card was updated)
And whenever the event goes to the queue, I would like to:
- send to a webhook
- send to a socket service
- send to an automatic actions service
every event should be processed in order (order matters) and also has to guarantee that the three queues events was processed
in this case, should I use consumer groups?
Also, is there a way I have multiple consumers for each consumer group?
ty
Amazing content. I had a few questions
1. Where does the broker keep the append only log file? Is it kept on disk or in memory ?
2. When the consumer tries to read the next message, does the broker do a sequential disk read (assuming it is on disk)?
Thanks!
very well explained!
ОтветитьWith idempotence enabled, acks=all is automatically applied, requiring acknowledgments from all in-sync replicas.
ОтветитьIs this bluescape where you are designing?
ОтветитьIs Zookeeper part of the Interview when discussing Kafka? I know Kafka can run without ZK these days but just wondering how much Zookeeper knowledge is required?
ОтветитьThanks. So what are the other 4 technologies in your top 5?
ОтветитьThis is super helpful. Thanks a million.
In the meantime, can you also shed light on the delivery mode of Kafka? In particular, how is the "exactly once" delivery achieved?
1. do consumers commit offset for every message they process or they can keep processing messages and commit offset once in a while.
2. would love to know about kafka streams and kafka channels with motivating examples :)
thanks @hellointerview & @evan!
Dumb question, but does Broker 2 need to have a leader for Topic A? Or does the Apache ZooKeeper (or some broker manager) manage partition follower replication between brokers?
ОтветитьThis is by far the best channel for discussing system design topics. My goodness, everything is structured so well. Well done!
ОтветитьI feel like half way you missed the topic a bit since Kafka is not really a message queue.
Ответитьabsolute best content. keep up with great work :)
Ответить!!!AWESOME!!! & THANK YOU
ОтветитьQuestion: How does Kafka ensure or maintain consistency between partitions (leader and followers)?
ОтветитьA Dead Letter Queue in Kafka is one or more Kafka topics that receive and store messages that could not be processed
Ответитьaim for 1 MB per message. And broker can take 1TB data and 10k messages per second. That would leave us with 100MB msg size which may not be a good practice. Instead , why not have 1M messages per second (given each msg of 1MB) ? cant we scale better this way ? in fact if msg size is small, we can bump up rate eg: 1kB msg (in case of some small text msgs like feeds or whatsapp) and 10^9 msgs per second.
ОтветитьPlease do one for Cassandra as well
ОтветитьThis is excellent
ОтветитьThis is an amazing explanation for a beginner like me in system design! I really appreciate the videos—please keep them coming. I’m hoping to crack into FAANG companies in the next few months.
ОтветитьBest video I have seen on the topic. Thank you!
ОтветитьThank you for the this, really enlighten me after using kafka in work for so long
ОтветитьHave you seen use cases with retention periods set to infinite and what would be some examples of that? retention ms and retention bytes both set to -1
ОтветитьThanks for this insightful session; it is incredibly informative, well structured and highly useful for a novice or an expert of Kafka. Your method of setting real life examples and ability to emulate consumer (not Kafka consumer 😊) mindset and going behind the reasoning is exemplary; great job 👍🏼
ОтветитьThis is such an awesome video. It was a good introduction and prompted me to then follow up on various details like how can i increase the paritition count, what are antipatterns, dynamically adding topics, adding brokers etc and other operational management considerations. This is just awesome and great job.
ОтветитьFor consumer retries, why use a Retry topic? why not put the message back into the main topic? Is the suggestion that when putting on the retry topic, maybe we also have a "number of retries" metadata which we keep incrementing (and the original producer doesnt need this). Or is there another reason? Say because maybe we want a separate consumer group to process retries for whatever reason?
ОтветитьQuick question: why use composite keys? Isn't it better to then just use option 1 (no keys)?
After all, you lose ordering anyway.
Awesome video!
Quick feedback: please zoom a bit more when there's text, it's a little hard to read 😅
The font itself is also a bit hard to read.
wish i had discovered this video earlier
ОтветитьOne of the best videos and explanation on the entire internet. Thanks a lot...
ОтветитьExceptionally insightful video. Nice dive-deep
ОтветитьGreat video on Kafka.
Question: It seems Kafka can do it all now. When do we still want to use Amazon SQS or RabbitMQ (use case) ?
Wrt. hot partitions, wouldn't it help to scale the consumer group? Or did I miss a constraint that wouldn't make this feasible?
ОтветитьA great video, thank you!
One thing that I would like to highlight is that the partitioning of the queue and the consumer groups are motivated by scaling, but topics is motivated by abstraction.
Ethan did mention this, but I think it was worth highlighting.
Haven't seen videos that explain things so clearly since Ali Ghodsi Data sciences lectures
ОтветитьThis is an amazing video!! What app are you using to document the details on the video?
ОтветитьThis is pure gold right here, my God! 🤯. Thanks for taking the time to break down these concepts.
ОтветитьThis is amazing content. The worth of these videos is in gold. Please keep these free always. Please dont ever pay-wall these videos. Thanks again. 🙏
Ответить