Dynamic Heartbeating Mechanism for Cluster Failover
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current High Availability (HA) cluster server products face delays and inconsistencies in responding to master node failures and network partitions due to the heartbeating method, which is not well-suited for large-scale clusters, leading to slower response times and potential false alerts.
Innovation Solution
The dynamic failover mechanism employs a selective heartbeating process that switches between all-active and distributed heartbeating modes based on cluster size, allowing multiple host computers to function as master nodes in small clusters and distributing roles in larger ones, reducing election delays and resource consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a single master node monitors all slave nodes in a cluster, then the heartbeating method works well for slave node failures, but the response time to master node failures becomes significantly delayed (30-45 seconds)
Solution Approach 1:
The patent merges the monitoring functions of multiple nodes into a collaborative system where each node can detect failures independently. Instead of relying on a single master node, all nodes participate in failure detection, eliminating the election delay and enabling immediate response to master node failures.
Solution Approach 2:
The patent implements preliminary action by having all nodes pre-configured with the ability to perform monitoring and failover operations. Each node maintains up-to-date inventory information and can immediately take action upon detecting a failure, without waiting for election or information collection.
2Device complexity
If a single master node maintains overall cluster inventory, then centralized control is achieved, but additional delay (5-10 seconds) occurs when a new master node must collect inventory information from all slave nodes
Solution Approach 1:
The patent segments the centralized inventory management into distributed inventory information. Each node maintains its own inventory data locally, eliminating the need for centralized collection and reducing the time required for inventory updates during failover events.
3Device complexity
If only one master node monitors heartbeats for all slave nodes, then the heartbeating method is simple to implement, but the method is not well suited for supporting large scale clusters
Solution Approach 1:
The patent segments the monitoring function by enabling multiple nodes to act as masters, each responsible for monitoring specific slave nodes. This distribution of monitoring responsibilities allows the system to scale to large clusters while maintaining the simplicity of the heartbeating mechanism.
Solution Approach 2:
The patent introduces dynamic role assignment where nodes can transition between slave and master roles based on cluster size and failure conditions. This dynamic adaptability allows the system to optimize its structure for different cluster scales while maintaining operational simplicity.
4Reliability
If network partition separates nodes into groups, then isolated groups must elect new master nodes, but this causes significant delay in responding to network partition conditions
Solution Approach 1:
The patent implements preliminary action by pre-configuring all nodes with the capability to function as masters and maintain inventory information. When network partition occurs, isolated groups can immediately form functional master nodes without election delays, as the capability is already in place.
5Reliability
If all nodes cannot communicate with each other in network isolation, then detection requires silent election and multiple timeouts, but this process is approximately 45 seconds slower than slave node failure detection
Solution Approach 1:
The patent enables nodes to self-detect network isolation conditions through local monitoring of communication status. Each node can independently determine when it is isolated from the network without requiring complex election processes or multiple timeout cycles, significantly reducing detection time.
Data Source
AI summary
System and method for providing failover protection for a cluster of host computers operates in one of two modes. When the number of host computers in the cluster is equal to or less than a threshold number, the host computers in the cluster operate in an all-active heartbeating mode in which each of the host computers is enabled to function as a master node. When the number of host computers in the cluster is greater than the threshold number, the host computers in the cluster operate in a distributed heartbeating mode in which some of the host computers function as master nodes and at least one of the host computers functions as a slave node.