cool hit counter Distributed Message Queues RocketMQ vs Kafka Architecture with Huge Differences No. 1_Intefrankly

Distributed Message Queues RocketMQ vs Kafka Architecture with Huge Differences No. 1

Original article from CSDN [travi]

After talking about kafka a few times and getting a good response, so this article, too, continues to talk about kafka learning and gives a good response

We know that in earlier versions of RocketMQ, there was a reliance on ZK. And in the current version, it is removing the dependency on ZK and moving to using the self-developed NameSrv.

And this NameSrv is stateless, you can deploy as many units as you want, and its code is very simple and very lightweight.

Then one wonders: ZooKeeper is a very common middleware used by the industry to manage clusters, for example kafka is dependent on ZK. So why should RocketMQ build its own wheels and do its own cluster management? Is it purely another Zookeeper?

This post attempts to illustrate why RocketMQ can de-ZK through a huge difference in architecture.

Kafka's architectural topology

We know that in Kafka, there are multiple partitions for 1 topic, and each partition has 1 master + multiple slaves. The correspondence is shown in the figure below.

note: There's only3 terrace robots(b0,b1,b2), each robots being the case thatmaster, alsoSlave。 specifically speaking, for example robotsb0, as far as sth is concernedpartition0 have one's say, It could beMaster; correspondspartition1 have one's say, It could be againSlave。

Architecture topology of RocketMQ

Unlike inside Kafka, a machine is both a Master and a Slave. Inside RocketMQ, 1 machine can only be either a Master or a Slave. This is set in stone inside the initial machine configuration. The architecture topology is as follows.

Here, the concept of queue in RocketMQ corresponds to partition in Kafka.

There are 3 Masters, 6 Slaves, that corresponds to physically, that's 3+6, that's 9 machines!!! Instead of the above like Kafka, 3 machines.

Master/Slave/Broker conceptual differences

With the 2 charts above, we can already visualize the huge difference between the 2. Reflecting on the concept, although the 2 have the 3 concepts of Master/Slave/Broker, their meanings are not the same.

Master/Slave Concept Differences

Kafka: Master/Slave is a logical concept, 1 machine, with both Master role and Slave role.

RocketMQ: Master/Slave It's a physical concept.,1 terrace robots, It can only beMaster perhapsSlave。 At the time of initial cluster configuration, appointed dead。 among othersMaster ofbroker id = 0,Slave ofbroker id > 0。

Broker conceptual differences

Kafka: Broker It's a physical concept.,1 sizebroker corresponds to1 terrace robots。

RocketMQ: Broker is a logical concept, 1 broker = 1 master + multiple slaves. That's why there are concepts like master broker, slave broker.

So here, how are master and slave paired? The answer is by broker name. A master and slave with the same broker name are paired.

Specifically inside the configuration, as follows.

// robots1 configuration brokerClusterName=DefaultCluster brokerName=broker-a brokerId=0 deleteWhen=04 fileReservedTime=48 brokerRole=ASYNC_MASTER flushDiskType=ASYNC_FLUSH

1 2 3 4 5 6 7 8 9

// robots2 configuration brokerClusterName=DefaultCluster brokerName=broker-a brokerId=1 deleteWhen=04 fileReservedTime=48 brokerRole=SLAVE flushDiskType=ASYNC_FLUSH

1 2 3 4 5 6 7 8

// robots3 configuration brokerClusterName=DefaultCluster brokerName=broker-a brokerId=2 deleteWhen=04 fileReservedTime=48 brokerRole=SLAVE flushDiskType=ASYNC_FLUSH

12 3 4 5 6 7 8

here robots1 harmony robots2, robots3 having the samebrokerName(broker-a), anbrokerId = 0, other2 sizebrokerId > 0。 consequently robots1 beMaster, robots2, 3 beSlave。

So here's what you can see: the definitions of RokcetMQ and Kafka regarding these 2 pairs of concepts are just the opposite! Kafka starts with a Broker and then produces a Master/Slave; RokcetMQ defines the Master/Slave first and then combines the Brokers.

Answer: why can you go to ZK?

The above comparison shows the difference between Kafka and RocketMQ on the 3 concepts of Master/Slave/Broker.

This difference also affects the mapping relationship between logical concepts like topic, partition and physical concepts like Master/Slave/Broker. Specifically.

Inside Kafka, the Maser/Slave is elected!!! RocketMQ doesn't need an election!!!

Inside Kafka, the Maser/Slave is elected!!! RocketMQ doesn't need an election!!!

Inside Kafka, the Maser/Slave is elected!!! RocketMQ doesn't need an election!!!

Three important things to say.。 specifically speaking, (located) atKafka interior,Master/Slave elections, have2 a step: (prefix indicating ordinal number, e.g. first, number two etc)1 a step, First throughZK In all robots in, Election of aKafkaController; (prefix indicating ordinal number, e.g. first, number two etc)2 a step, And then by thisController, Decides that eachpartition ofMaster who is it?,Slave who is it?。

The Master/Slave here is dynamic, i.e.: when the Master hangs, 1 Slave will switch to the Master.

In RocketMQ, on the other hand, no election is required and the roles of Master/Slave are fixed. When a Master hangs, you can write to other Masters, but it doesn't say that a Slave switches to a Master.

This simplification, feasibleRocketMQ Can not rely onZK Just good managementTopic/queue and physical robots The mapping relationship between the, High availability is also achieved。

Here, it also involves the problem of "message order" that I mentioned in the previous article: in Kafka, a partition must have a strict mapping relationship with a Master, and if the Master hangs, a Master has to be elected from other Slave; while in RocketMQ, this restriction is released, and if the Master corresponding to a queue hangs, it will cut to other Masters, instead of having to elect one.

With that said, the answer is basically known: RocketMQ doesn't need to have heavy election logic like Kafka does, it simplifies the problem. All that's left is the routing information for the topic/queue, and that's taken care of with a simple NameServer that's lightweight, stateless, and has a good guarantee of reliability.

The creation process of Topic

Here's a look at what parameters need to be specified for each of Kafka and RocketMQ when creating a topic, from a usage perspective

It is also clear from these parameters that,2 non-governmental organizationstopic, partition This logical concept and physical robots The mapping relationship between, It's very different.。

RocketMQ commands for creating a topic

The following code comes from the class UpdateTopicSubCommand, which is the class that Rocketmq calls when it creates a topic. Here are a few key parameters, others I have omitted.


c: //b harmonyc2 choose1,b is designatedtopic whereabouts robots,c is designatedtopic whereaboutscluster

topic: //This is a basic parameter, nothing to talk about

readQueueNums/writeQueueNums: // Number of queues. The default 2 are equal and are 8. About this readQueueNums/writeQueueNums, it is a RocketMQ-specific concept, which will be analyzed in detail later. Here it is assumed that they are 2 equal and the same 1.

Option opt = new Option("b", "brokerAddr", true, "create topic to which broker"); opt.setRequired(false); options.addOption(opt); opt = new Option("c", "clusterName", true, "create topic to which cluster"); opt.setRequired(false); options.addOption(opt); opt = new Option("t", "topic", true, "topic name"); opt.setRequired(true); options.addOption(opt); opt = new Option("r", "readQueueNums", true, "set read queue nums"); opt.setRequired(false); options.addOption(opt); opt = new Option("w", "writeQueueNums", true, "set write queue nums"); opt.setRequired(false); options.addOption(opt); 。。。 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

Kafka commands for creating a topic

Compared to RocketMQ, there are 2 of the same parameters: 1 is the topic and one is the number of queues, which is the -partitions here.

bin/ --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic 1

One significant difference between the 2 when creating a topic

Kafka has a parameter replication-factor, which means that it specifies how many Slave are assigned to 1 Master?

RocketMQ has a parameter c, the clusterName, to specify the pairing of all Masters and Slaves inside this cluster (multiple masters, multiple slaves) corresponding to the same 1 topic!!!

By default, allMaster harmonySlave belong to the same1 cluster, Also known as the above3 terrace robots The configuration of the first1 individual parameter:brokerClusterName=DefaultCluster。

Combined with the architecture topology diagram above, we can see that

as far as sth is concernedkafka have one's say, You specify.topic, itspartition number of cases, itsmaster/slave proportion, The system then automatically removes all the data from all robots in, for eachtopic_partition allocate1 sizemaster + severalslave;

For RokcetMQ, you specify the topic, the number of queues it has, and the cluster it corresponds to. Then the system automatically establishes the mapping between this cluster (multiple masters + multiple slaves) and your topic.

2、Obscure operations Tensormaxgather that pytorch newbies need to be aware of
3、Sharpshooter Series Terminal split screen sharpshooter tmux
4、Replacing reactloadable with ReactSuspense
5、Big Data 247Turkeys national database of 50 million peoples privacy compromisedMedical data service merchant RenMed has secured 10 million in funding

    已推荐到看一看 和朋友分享想法
    最多200字,当前共 发送