Kafka partition leader replica election method and apparatus, device, medium and product
By dividing the Kafka cluster into partition groups and adopting a partition group election strategy, the problems of excessive Zookeeper load and network congestion in traditional technologies are solved, achieving more efficient partition master replica election and improving the stability and performance of the Kafka cluster.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- CHINA TELECOM CLOUD TECH CO LTD
- Filing Date
- 2025-11-19
- Publication Date
- 2026-06-04
AI Technical Summary
In a Kafka cluster, traditional partition master-replica election can lead to excessive Zookeeper load and network congestion when dealing with tens of thousands of partitions.
By dividing the partition group into multiple partition groups and registering partition group-level listeners in Zookeeper, a partition group election strategy is adopted to elect the master partition group, thereby reducing Zookeeper load and improving network congestion.
It effectively reduced the load on Zookeeper, improved the efficiency of partition master replica election, reduced network congestion, and enhanced the stability and performance of the Kafka cluster.
Smart Images
Figure CN2025135968_04062026_PF_FP_ABST
Abstract
Description
Methods, apparatus, devices, media, and products for electing Kafka partition primary replicas.
[0001] Related applications
[0002] This application claims priority to Chinese patent application filed on November 29, 2024, with application number 202411732393.5, entitled "Method and apparatus, device, media and product for electing primary replicas of Kafka partitions", the entire contents of which are incorporated herein by reference. Technical Field
[0003] This application relates to the field of Kafka cluster technology, and in particular to a method, apparatus, device, medium and product for electing a primary replica of a Kafka partition. Background Technology
[0004] Kafka is a distributed message streaming platform that supports multiple partitions, multiple replicas, and coordination based on Zookeeper. It is also an open-source message engine system based on the publish-subscribe pattern.
[0005] Kafka partition leader election typically occurs when the leader of a partition fails. That is, when the leader of one or more partitions in a Kafka cluster fails, a new leader needs to be elected from the partition's followers.
[0006] In traditional technologies, when Kafka has more than 10,000 partitions, the election of the primary replica of the partitions can lead to problems such as excessive load on Zookeeper and network congestion. Summary of the Invention
[0007] According to various embodiments of this application, a method, apparatus, device, medium, and product for electing a primary replica of a Kafka partition are provided.
[0008] In a first aspect, embodiments of this application provide a method for electing a primary replica of a Kafka partition, the method comprising:
[0009] If it is determined that a partition master copy election is required, determine whether the partition to which the master copy election is required belongs to a partition group;
[0010] If it exists, a primary partition group is elected from the partition groups among multiple server nodes according to the preset partition group election strategy; and
[0011] Write the partition group information of the elected primary partition group to Zookeeper, so that Zookeeper can instruct the Kafka controller to update the partition group information of the primary partition group stored in the Kafka controller. After the update, the Kafka controller sends the first change request to the server node where the elected primary partition group is located.
[0012] The first change request is used to request the server node where the elected primary partition group is located to change each secondary replica of the elected primary partition group to a primary replica.
[0013] In one embodiment, the Kafka cluster includes multiple topics, each topic including multiple partitions distributed across different server nodes; each partition including multiple replicas.
[0014] In one embodiment, before electing a primary partition group from the partition groups among multiple server nodes according to a preset partition group election strategy, the method further includes:
[0015] Determine whether each replica of the partition group in the multiple server nodes is valid; and
[0016] If all slave replicas of the partition group in each server node are valid, then the step of electing a primary partition group from the partition groups in multiple server nodes according to a preset partition group election strategy is executed.
[0017] In one embodiment, the method further includes:
[0018] If at least one replica is invalid, the server node containing the invalid replica is removed from the plurality of server nodes. After removal, a primary partition group is elected from the remaining server nodes according to a preset partition group election strategy.
[0019] If each partition group in each server node has at least one invalid slave replica, the election ends.
[0020] In one embodiment, the method further includes:
[0021] If Zookeeper detects that at least one of the multiple server nodes has crashed, it determines that a partition master replica election is required.
[0022] In one embodiment, the method further includes:
[0023] If the partition requiring primary replica election does not belong to a partition group, a primary replica is elected from the replicas among multiple server nodes according to the preset partition election strategy; and
[0024] The partition information of the elected primary replica is written to Zookeeper, so that Zookeeper can instruct the Kafka controller to update the partition information of the primary replica stored in the Kafka controller. After the update, the Kafka controller sends a second change request to the server node where the elected primary replica is located.
[0025] The second change request is used to request the server node where the elected primary replica is located to change the elected primary replica from a secondary replica to a primary replica.
[0026] In one embodiment, if the partition that needs to perform master replica election has a partition group to which it belongs, then in the plurality of server nodes, each of the partition groups in the server node registers a partition group-level listener in Zookeeper;
[0027] If the partition requiring master replica election does not belong to a partition group, then among the multiple server nodes, each server node's slave replica registers a partition-level listener in ZooKeeper.
[0028] Secondly, embodiments of this application provide a Kafka partition primary replica election device, the device comprising:
[0029] The group election module is used to determine whether the partition to be elected as the primary replica needs to be elected has a corresponding partition group when it is determined that a primary replica election needs to be performed. If so, a primary partition group is elected from the partition groups among multiple server nodes according to a preset partition group election strategy.
[0030] The update and change module is used to write the partition group information of the elected primary partition group to Zookeeper, so that Zookeeper can instruct the Kafka controller to update the partition group information of the primary partition group stored in the Kafka controller. After the update, the Kafka controller sends the first change request to the server node where the elected primary partition group is located.
[0031] The first change request is used to request the server node where the elected primary partition group is located to change each secondary replica of the elected primary partition group to a primary replica.
[0032] In one embodiment, the group election module is further configured to determine whether each replica of the partition group in the multiple server nodes is valid before electing a master partition group from the partition groups in the multiple server nodes according to a preset partition group election strategy; and if each replica of the partition group in the multiple server nodes is valid, then execute the step of electing a master partition group from the partition groups in the multiple server nodes according to the preset partition group election strategy.
[0033] In one embodiment, the group election module is further configured to: if at least one slave replica is invalid, remove the server node containing the invalid slave replica from the plurality of server nodes; and after removal, elect a master partition group from the partition groups in the remaining server nodes according to a preset partition group election strategy; and if each partition group in each server node has at least one invalid slave replica, end the election.
[0034] In one embodiment, the group election module is further configured to determine that a partition master replica election is required when Zookeeper detects that at least one of the plurality of server nodes has crashed.
[0035] In one embodiment, the group election module is further configured to, if the partition requiring primary replica election does not belong to a partition group, elect a primary replica from among the secondary replicas of multiple server nodes according to a preset partition election strategy; and
[0036] The update and change module is further configured to write the partition information of the elected primary replica into Zookeeper, so as to instruct the Kafka controller to update the partition information of the primary replica stored in the Kafka controller through Zookeeper. After the update, the Kafka controller sends a second change request to the server node where the elected primary replica is located. The second change request is used to request the server node where the elected primary replica is located to change the elected primary replica from a secondary replica to a primary replica.
[0037] Thirdly, embodiments of this application provide a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.
[0038] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in the first aspect.
[0039] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.
[0040] Details of one or more embodiments of this application are set forth in the following drawings and description. Other features, objects, and advantages of this application will become apparent from the specification, drawings, and claims. Attached Figure Description
[0041] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the disclosed drawings without creative effort.
[0042] Figure 1 is a flowchart illustrating the election method for the primary replica of a Kafka partition in some embodiments;
[0043] Figure 2 is one of the schematic diagrams of a Kafka cluster in some embodiments;
[0044] Figure 3 is a second schematic diagram of a Kafka cluster in some embodiments;
[0045] Figure 4 is a schematic diagram of a Kafka cluster in one of several embodiments;
[0046] Figure 5 is one of the flowcharts illustrating the Kafka partition primary replica election method in some embodiments;
[0047] Figure 6 is a flowchart illustrating the second part of the Kafka partition primary replica election method in some embodiments;
[0048] Figure 7 is a block diagram of the election device for the primary replica of a Kafka partition in some embodiments. Detailed Implementation
[0049] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.
[0051] It is understood that the terms "first," "second," etc., used herein may be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of this application, a first resistor may be referred to as a second resistor, and similarly, a second resistor may be referred to as a first resistor. Both the first resistor and the second resistor are resistors, but they are not the same resistor.
[0052] It is understood that the term "connection" in the following embodiments should be understood as "electrical connection," "communication connection," etc., if the connected circuits, modules, units, etc., have electrical signal or data transmission with each other.
[0053] It is understandable that "at least one" refers to one or more, and "multiple" refers to two or more. "At least a part of an element" refers to part or all of an element.
[0054] When used herein, the singular forms of “a,” “an,” and “the” may also include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising / including” or “having,” etc., specify the presence of the stated features, wholes, steps, operations, components, parts, or combinations thereof, but do not preclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts, or combinations thereof. Meanwhile, the term “and / or” as used in this specification includes any and all combinations of the associated listed items.
[0055] In one example embodiment, referring to Figure 1, a method for electing a primary replica of a Kafka partition is provided, which may include the following steps 102-106.
[0056] 102. If it is determined that a partition primary replica election is required, determine whether the partition to which the primary replica election is required has a corresponding partition group.
[0057] A Kafka cluster can include multiple topics, each topic can include multiple partitions, and these partitions can be distributed across different server nodes (brokers); each partition can include multiple replicas.
[0058] For example, referring to Figure 2, a topic in a Kafka cluster comprises nine partitions, namely Partition1 to Partition9. Each of these nine partitions includes four replicas, distributed across Broker1 to Broker4. Taking Partition1 as an example: Partition1 includes four replicas, distributed across Broker1 to Broker4. The four replicas of Partition1 include Partition1 itself; in other words, Partition1 itself is considered one of the four replicas of Partition1. Partition1L is the elected primary replica among the four replicas of Partition1, and Partition1F is the secondary replica among the four replicas of Partition1.
[0059] For example, referring to Figure 3, in Broker1 to Broker4, Partition1, Partition4 and Partition7 all belong to a partition group, and the partition group they belong to is PartitionGroup1; Partition8 and Partition9 both belong to a partition group, and the partition group they belong to is PartitionGroup2; Partition2, Partition3, Partition5 and Partition6 do not belong to a partition group, that is, they are not joined to a partition group.
[0060] Referring to Figure 3, we can see that for PartitionGroup1, PartitionGroup1 exists in Broker1 through Broker4. When PartitionGroup1 in Broker1 is the primary partition group (PartitionGroup1L) and PartitionGroup1 in Broker2 through Broker4 is a secondary partition group (PartitionGroup1F), the partitions of PartitionGroup1L are all primary replicas, namely Partition1L, Partition4L, and Partition7L, and the partitions of PartitionGroup1F are all secondary replicas, namely Partition1F, Partition4F, and Partition7F. The same applies to PartitionGroup2. In other words, each partition in the primary partition group is a primary replica, and each partition in the secondary partition group is a secondary replica. The primary / secondary status of each partition in the partition group is consistent with the primary / secondary status of the partition group itself.
[0061] 104. If it exists, then a primary partition group is elected from the partition groups of multiple server nodes according to the preset partition group election strategy.
[0062] For example, referring to Figure 4, if Broker1 fails, all partitions in Broker1 will become invalid. The primary replicas of Partition1, Partition4, and Partition7 are all on Broker1, as are the primary replicas of Partition8 and Partition9. If Broker1 fails, Partition1L, Partition4L, Partition7L, Partition8L, and Partition9L will all become invalid, thus requiring a primary replica election for Partition1, Partition4, Partition7, Partition8, and Partition9. However, the primary replicas of Partition2, Partition3, Partition5, and Partition6 are not on Broker1. If Broker1 fails, Partition2F, Partition3F, Partition5F, and Partition6F in Broker1 will become invalid, but a primary replica election for Partition2, Partition3, Partition5, and Partition6 is not required.
[0063] Referring to Figure 4, when performing primary replica election for Partition 1, Partition 4, Partition 7, Partition 8, and Partition 9, since Partition 1, Partition 4, and Partition 7 all belong to Partition Group 1, one of the three can be elected as the primary partition group according to the preset partition group election strategy. For example, Partition Group 1 in Broker 2 can be elected as the primary partition group, while Partition Group 1 in Broker 3 and Partition Group 1 in Broker 4 can both be elected as secondary partition groups.
[0064] Referring to Figure 4, since Partition 8 and Partition 9 both belong to Partition Group 2, one of the three can be elected as the primary partition group according to the preset partition group election strategy: Partition Group 2 in Broker 2, Partition Group 2 in Broker 3, and Partition Group 2 in Broker 4. For example, Partition Group 2 in Broker 4 can be elected as the primary partition group, while Partition Group 2 in Broker 2 and Partition Group 2 in Broker 3 can both be elected as secondary partition groups.
[0065] 106. Write the partition group information of the elected primary partition group to Zookeeper, so that Zookeeper can instruct the Kafka controller to update the partition group information of the primary partition group stored in the Kafka controller. After the update, the Kafka controller sends a first change request to the server node where the elected primary partition group is located. The first change request is used to request the server node where the elected primary partition group is located to change each replica of the elected primary partition group to a primary replica.
[0066] For example, continuing to refer to Figure 4, the partition group information of partition group 1 in the elected Broker2 is written to ZooKeeper, and the partition group information of partition group 2 in the elected Broker4 is also written to ZooKeeper. After writing, ZooKeeper instructs the Kafka Controller to replace the partition group information of partition group 1 in Broker1 stored in the Kafka Controller with the partition group information of partition group 1 in Broker2, and to replace the partition group information of partition group 2 in Broker1 stored in the Kafka Controller with the partition group information of partition group 2 in Broker4. After the replacement, that is, after the update, the Kafka Controller sends a first change request to Broker2, which requests Broker2 to change all slave replicas of partition group 1 in Broker2 to primary replicas. The Kafka Controller also sends another first change request to Broker4, which requests Broker4 to change all slave replicas of partition group 2 in Broker4 to primary replicas.
[0067] The above is just an example. Make sure the number of partition groups in the Broker is less than the number of partitions in the Broker. In other words, the number of partitions in a partition group should be multiple. It can be understood that the more partitions in a partition group, the greater the reduction in load on Zookeeper.
[0068] This application's embodiment first elects a primary partition group by dividing the system into partition groups containing multiple partitions, and then directly obtains the elected primary replica based on the primary partition group. During the election, only the partition group needs to be selected, which significantly reduces ZooKeeper's load compared to traditional techniques that elect a primary replica based on a single partition. This also improves network congestion and election efficiency.
[0069] In an exemplary embodiment, referring to FIG5, before electing a primary partition group from the partition groups among multiple server nodes according to a preset partition group election strategy in step 104, the Kafka partition primary replica election method may further include the following steps 502 to 508.
[0070] 502, Determine whether each replica of the partition group in the multiple server nodes is valid.
[0071] For example, continuing to refer to Figure 4, it is determined whether each slave copy of partition group 1 in Broker2, each slave copy of partition group 1 in Broker3, and each slave copy of partition group 1 in Broker4 are all valid; and it is determined whether each slave copy of partition group 2 in Broker2, each slave copy of partition group 2 in Broker3, and each slave copy of partition group 2 in Broker4 are all valid.
[0072] 504. If all slave replicas of the partition group in each server node are valid, then the step of electing a primary partition group from the partition groups in multiple server nodes according to a preset partition group election strategy is executed.
[0073] For example, continuing to refer to Figure 4, if all slave replicas of partition group 1 in Broker2, partition group 1 in Broker3, and partition group 1 in Broker4 are valid, then step 104 is executed, in which primary replica election is performed for Partition 1, Partition 4, and Partition 7 in partition group 1 of Broker2, partition group 1 of Broker3, and partition group 1 of Broker4; and if all slave replicas of partition group 2 in Broker2, partition group 2 in Broker3, and partition group 2 in Broker4 are valid, then step 104 is executed, in which primary replica election is performed for Partition 8 and Partition 9 in partition group 2 of Broker2, partition group 2 of Broker3, and partition group 2 of Broker4.
[0074] 506. If at least one replica is invalid, the server node containing the invalid replica is removed from the plurality of server nodes. After removal, a primary partition group is elected from the partition groups of the remaining server nodes according to a preset partition group election strategy.
[0075] For example, continuing to refer to Figure 4, if at least one slave replica of partition group 1 in Broker3 is invalid, then in partition group 1 of Broker2 and partition group 1 of Broker4, primary replica election is performed for Partition1, Partition4, and Partition7, in which Broker3 is removed. If at least one slave replica of partition group 2 in Broker2 is invalid, and at least one slave replica of partition group 2 in Broker3 is invalid, then in partition group 2 of Broker4, primary replica election is performed for Partition8 and Partition9, in which Broker2 and Broker3 are removed. In fact, since only partition group 2 in Broker4 remains at this time, partition group 2 in Broker4 can be directly used as the primary replica group for Partition8 and Partition9.
[0076] 508. If each partition group in each server node has at least one invalid slave replica, then the election ends.
[0077] For example, continuing to refer to Figure 4, if at least one slave copy of partition group 1 in Broker2, at least one slave copy of partition group 1 in Broker3, and at least one slave copy of partition group 1 in Broker4 are all invalid, the election ends, and the election of primary copies for Partition1, Partition4, and Partition7 is withdrawn. If at least one slave copy of partition group 2 in Broker2, at least one slave copy of partition group 2 in Broker3, and at least one slave copy of partition group 2 in Broker4 are all invalid, the election ends, and the election of primary copies for Partition8 and Partition9 is withdrawn.
[0078] In an exemplary embodiment, the Kafka partition primary replica election method may further include: determining that a partition primary replica election is required when Zookeeper detects that at least one of the plurality of server nodes has crashed.
[0079] In an exemplary embodiment, referring to FIG6, the method for electing the primary replica of a Kafka partition may further include the following steps 108 to 110.
[0080] 108. If the partition that needs to elect a primary replica does not belong to a partition group, then the primary replica is elected from the replicas of multiple server nodes according to the preset partition election strategy.
[0081] For example, continuing to refer to Figure 3, Partition 2, Partition 3, Partition 5, and Partition 6 do not belong to any partition groups, meaning they are not joined to any partition groups. The primary replicas of Partition 2 and Partition 5 are both on Broker 2. Therefore, if Broker 2 fails, the primary replica of Partition 2 can be elected from Partition 2F in Broker 1, Partition 2F in Broker 3, and Partition 2F in Broker 4. Similarly, if Broker 2 fails, the primary replica of Partition 5 can be elected from Partition 5F in Broker 1, Partition 5F in Broker 3, and Partition 5F in Broker 4. Since the primary replicas of partitions other than Partition 2 and Partition 5 are not on Broker 2 when Broker 2 fails, there is no need to elect a primary replica for these partitions.
[0082] 110. Write the partition information of the elected primary replica to Zookeeper so that Zookeeper can instruct the Kafka controller to update the partition information of the primary replica stored in the Kafka controller. After the update, the Kafka controller sends a second change request to the server node where the elected primary replica is located. The second change request is used to request the server node where the elected primary replica is located to change the elected primary replica from a secondary replica to a primary replica.
[0083] For example, continuing to refer to Figure 3, the partition information of the primary replica of the elected Partition 2 is written to ZooKeeper, and the partition information of the primary replica of the elected Partition 5 is also written to ZooKeeper. After writing, ZooKeeper instructs the Kafka controller to replace the partition information of Partition 2 in Broker 2 stored in the Kafka controller with the partition information of the primary replica of the elected Partition 2, and to replace the partition information of Partition 5 in Broker 2 stored in the Kafka controller with the partition information of the primary replica of the elected Partition 5. After the replacement, that is, after the update, the Kafka controller sends a second change request to the Broker where the primary replica of the elected Partition 2 is located. This second change request is used to request the Broker to complete the change from a replica to a primary replica of the elected Partition 2. Similarly, the same applies to Partition 5.
[0084] In one exemplary embodiment, if the partition to which master replica election is required has a partition group to which it belongs, then in the plurality of server nodes, each server node has a partition group-level listener registered in ZooKeeper for the partition group; if the partition to which master replica election is required does not have a partition group to which it belongs, then in the plurality of server nodes, each server node has a slave replica registered in ZooKeeper for a partition-level listener.
[0085] Since this application uses listeners at each partition group level to listen to each partition group separately, compared to the traditional technology where each partition-level listener listens to each partition, and each listener acts as a load on Zookeeper, this greatly reduces the load on Zookeeper, improves network congestion, and increases election efficiency.
[0086] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0087] Based on the same inventive concept, this application also provides a Kafka partition primary replica election apparatus for implementing the Kafka partition primary replica election method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations of one or more Kafka partition primary replica election apparatus embodiments provided below can be found in the limitations of the Kafka partition primary replica election method described above, and will not be repeated here.
[0088] In an exemplary embodiment, as shown in Figure 7, a Kafka partition primary replica election device is provided, comprising:
[0089] The group election module 710 is used to determine whether the partition to be elected as the primary replica needs to be elected has a corresponding partition group when it is determined that a primary replica election needs to be performed. If it does, a primary partition group is elected from the partition groups among multiple server nodes according to a preset partition group election strategy.
[0090] The update and change module 720 is used to write the partition group information of the elected primary partition group to Zookeeper, so as to instruct the Kafka controller to update the partition group information of the primary partition group stored in the Kafka controller through Zookeeper. After the update, the Kafka controller sends the first change request to the server node where the elected primary partition group is located.
[0091] The first change request is used to request the server node where the elected primary partition group is located to change each secondary replica of the elected primary partition group to a primary replica.
[0092] In an exemplary embodiment, the group election module 710 is further configured to determine whether each replica of the partition group in the multiple server nodes is valid before electing a primary partition group from the partition groups in the multiple server nodes according to a preset partition group election strategy; if each replica of the partition group in the multiple server nodes is valid, then the step of electing a primary partition group from the partition groups in the multiple server nodes according to the preset partition group election strategy is executed.
[0093] In an exemplary embodiment, the group election module 710 is further configured to, if at least one slave replica is invalid, remove the server node containing the invalid slave replica from the plurality of server nodes, and after removal, elect a primary partition group from the partition groups in the remaining server nodes according to a preset partition group election strategy; if each partition group in each server node has at least one invalid slave replica, then the election ends.
[0094] In an exemplary embodiment, the group election module 710 is further configured to determine that a partition master replica election is required when Zookeeper detects that at least one of the plurality of server nodes has crashed.
[0095] In an exemplary embodiment, the group election module 710 is further configured to, if the partition to which the primary replica election is required does not belong to a partition group, elect a primary replica from the secondary replicas among multiple server nodes according to a preset partition election strategy.
[0096] In an exemplary embodiment, the update / change module 720 is further configured to write the partition information of the elected primary replica into Zookeeper, so as to instruct the Kafka controller to update the partition information of the primary replica stored in the Kafka controller through Zookeeper, and after the update, the Kafka controller sends a second change request to the server node where the elected primary replica is located; wherein, the second change request is used to request the server node where the elected primary replica is located to realize the change of the elected primary replica from a secondary replica to a primary replica.
[0097] In one exemplary embodiment, if the partition to which master replica election is required has a partition group to which it belongs, then in the plurality of server nodes, each server node registers a partition group-level listener for the partition group in ZooKeeper; if the partition to which master replica election is required does not have a partition group to which it belongs, then in the plurality of server nodes, each server node's slave replica registers a partition-level listener in ZooKeeper.
[0098] The aforementioned Kafka partition primary replica election mechanism can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.
[0099] In an exemplary embodiment, a computer device is provided, which may be a server. The computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is connected to the system bus via the I / O interfaces. The processor of the diagnostic instrument provides computing and control capabilities. The memory of the diagnostic instrument includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the diagnostic instrument stores N configuration code data. The I / O interfaces of the diagnostic instrument are used for exchanging information between the processor and external devices. The communication interface of the diagnostic instrument is used for communicating with an external terminal via a network connection. When the computer program is executed by the processor, it implements the steps of any of the above method embodiments.
[0100] In one exemplary embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described embodiments of the method for electing a primary replica of any Kafka partition.
[0101] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the above-described embodiments of the method for electing a primary replica of any Kafka partition.
[0102] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.
[0103] In the description of this specification, references to terms such as "some embodiments," "other embodiments," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative descriptions of the above terms do not necessarily refer to the same embodiments or examples.
[0104] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0105] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A method for electing a Kafka partition primary replica, characterized in that, The method comprises: In the case of determining that partition leader election is needed, judging whether the partition needing leader election exists a subordinate partition group; If yes, according to a preset partition group election strategy, a master partition group is elected from the partition group in the plurality of server nodes; and The partition group information of the elected master partition group is written into Zookeeper, so as to instruct the Kafka controller to update the partition group information of the master partition group stored in the Kafka controller through Zookeeper, and after the update, the Kafka controller sends a first change request to the server node where the elected master partition group is located; Wherein, the first change request is used to request the server node where the elected master partition group is located, so that each slave replica of the elected master partition group is changed to a master replica.
2. The method of claim 1, wherein, The Kafka cluster comprises a plurality of topics, the topics comprise a plurality of partitions, and the plurality of partitions are distributed in different server nodes; the partition comprises a plurality of replicas.
3. The method of claim 1, wherein, Before the master partition group is elected from the partition group in the plurality of server nodes according to the preset partition group election strategy, the method further comprises: Judging whether each slave replica of the partition group in the plurality of server nodes is valid; and If each slave replica of the partition group in each server node is valid, the step of electing the master partition group from the partition group in the plurality of server nodes according to the preset partition group election strategy is executed.
4. The method of claim 3, wherein, The method further comprises: If at least one slave replica is invalid, the server node where the invalid slave replica is located is excluded from the plurality of server nodes, and after the exclusion, the master partition group is elected from the partition group in the remaining server nodes according to the preset partition group election strategy; and If at least one invalid slave replica exists in each server node, the election is ended.
5. The method of claim 1, wherein, The method further comprises: In the case that at least one server node in the plurality of server nodes is down, it is determined that partition leader election is needed.
6. The method of claim 1, wherein, The method further comprises: If the partition needing leader election does not exist a subordinate partition group, a master replica is elected from the slave replicas in the plurality of server nodes according to a preset partition election strategy; and The partition information of the elected master replica is written into Zookeeper, so as to instruct the Kafka controller to update the partition information of the master replica stored in the Kafka controller through Zookeeper, and after the update, the Kafka controller sends a second change request to the server node where the elected master replica is located; Wherein, the second change request is used to request the server node where the elected master replica is located, so that the elected master replica is changed from a slave replica to a master replica.
7. The method of claim 6, wherein, In the case that the partition needing leader election exists a subordinate partition group, in the plurality of server nodes, the partition group in each server node is registered with a partition group level listener in Zookeeper; In a case where the partition for which the leader election is required does not exist in the partition group to which the partition belongs, each of the slave replicas in the server nodes registers a partition-level listener in the Zookeeper.
8. An apparatus for electing a Kafka partition primary replica, the apparatus comprising: The apparatus comprises: The grouping election module is configured to, in a case where it is determined that the leader election is required, determine whether the partition for which the leader election is required exists in the partition group to which the partition belongs, and if yes, elect a master partition group from the partition group in the plurality of server nodes according to a preset partition group election strategy. The update changing module is configured to write the partition group information of the elected master partition group into the Zookeeper, so as to instruct the Kafka controller to update the partition group information of the master partition group stored in the Kafka controller through the Zookeeper, and after the update, send a first changing request to the server node where the elected master partition group is located. The first changing request is configured to request the server node where the elected master partition group is located to change each slave replica of the elected master partition group into a leader replica.
9. The apparatus according to claim 8, wherein The grouping election module is further configured to, before the step of electing the master partition group from the partition group in the plurality of server nodes according to the preset partition group election strategy, determine whether each slave replica of the partition group in the plurality of server nodes is valid, and if yes, execute the step of electing the master partition group from the partition group in the plurality of server nodes according to the preset partition group election strategy.
10. The apparatus according to claim 8, wherein The grouping election module is further configured to, if at least one slave replica is invalid, exclude the server node where the invalid slave replica is located from the plurality of server nodes, and after the exclusion, elect the master partition group from the partition group in the remaining server nodes according to the preset partition group election strategy, and if each of the partition groups in the plurality of server nodes has at least one invalid slave replica, end the election.
11. The apparatus according to claim 8, wherein The grouping election module is further configured to, in a case where the Zookeeper detects that at least one of the plurality of server nodes is down, determine that the leader election is required.
12. The apparatus according to claim 8, wherein The grouping election module is further configured to, if the partition for which the leader election is required does not exist in the partition group to which the partition belongs, elect a leader replica from the slave replicas in the plurality of server nodes according to a preset partition election strategy, and The update change module is further configured to write the partition information of the elected primary replica into a Zookeeper, to instruct a Kafka controller to update the partition information of the primary replica stored in the Kafka controller through the Zookeeper, and after the update, the Kafka controller sends a second change request to a server node where the elected primary replica is located; the second change request is configured to request the server node where the elected primary replica is located to implement the change of the elected primary replica from a slave replica to a primary replica.
13. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, implements the method of any one of claims 1-7.
14. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor implements the method of any one of claims 1-7 when executing the computer program.
15. A computer program product comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the method of any one of claims 1-7. The computer program, when executed by a processor, implements the method of any one of claims 1-7.