Distributed Cluster Master Node Failure Detection via Reference Node
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current high availability distributed clusters have a lengthy failure detection time for the master node, as Slave nodes only determine the master node is faulty after multiple missed heartbeat messages, leading to delayed election of a new master node.
Innovation Solution
Incorporating a reference node and a secondary master node, where the secondary node determines the master node's disconnection by monitoring heartbeat messages from both the reference node and itself, allowing for rapid detection of the master node's faultiness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If Slave nodes perform multiple missed heartbeat message detections before determining Master node failure, then false positives are reduced, but failure detection time becomes excessively long
Solution Approach 1:
A witness node is introduced as an intermediary to monitor the Master node's heartbeat messages and provide testimony about the Master node's status. When the witness node detects that the Master node has stopped sending heartbeat messages, it immediately notifies the Slave nodes, enabling rapid failure detection without requiring Slave nodes to wait for multiple missed heartbeats, thus resolving the contradiction between detection accuracy and detection time
Solution Approach 2:
The witness node performs preliminary monitoring of the Master node's heartbeat messages and prepares failure notification in advance. When the Master node fails, the witness node has already detected the absence of heartbeat messages and can immediately trigger the failure notification mechanism, eliminating the delay that would otherwise require Slave nodes to perform multiple detection cycles before declaring failure
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Embodiments of the present invention provide a method and a device for processing a failure in at least one distributed cluster, and a system. The at least one distributed cluster includes a first distributed cluster. The first distributed cluster includes a first Master node, a first Slave node, a first reference node, and a first secondary node that serves as a backup of the first Master node. The first secondary node receives a heartbeat message that is sent by the first reference node and that includes first indication information that is used to indicate that the first reference node is disconnected from the first Master node. The first secondary node determines, according to the first indication information, that the first reference node is disconnected from the first Master node. When it is detected that a heartbeat message sent by the first Master node to the first secondary node is interrupted, the first secondary node determines that the first secondary node is also disconnected from the first Master node. The first secondary node determines that the first Master node is faulty. In the embodiments of the present invention, a failure detection time can be effectively shortened.