Dynamic Heartbeat Timeout for Cluster Split Brain Prevention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In ACT-SBY cluster configurations, the occurrence of split brain due to communication failures in heartbeat packets leads to inconsistent data and service disruptions, making it challenging to maintain high reliability and availability, especially in carrier-oriented systems.
Innovation Solution
A server apparatus with a heartbeat transmission/reception part and a corresponding node monitoring part that dynamically adjusts the timeout period based on the reception state of heartbeat packets, preventing erroneous failover and maintaining service integrity by ensuring accurate node failure detection and minimizing service impact.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a fixed timeout period is used for heartbeat packet monitoring, then the system structure is simple and easy to implement, but the system reliability deteriorates due to split brain occurrences during network communication failures
Solution Approach 1:
The timeout period is changed from a fixed value to a dynamic value that adjusts based on the reception state of heartbeat packets. When heartbeat packets are received normally, the timeout period remains at a standard value. When packet loss or delay is detected, the timeout period is extended automatically. This dynamic adjustment prevents erroneous failover during network issues while maintaining simple operation during normal conditions.
Solution Approach 2:
The monitoring mechanism incorporates feedback by continuously observing the reception state of heartbeat packets and using this information to adjust the timeout period. The system monitors whether heartbeat packets are received within expected timeframes and modifies the timeout threshold accordingly, creating a closed-loop control system that adapts to network conditions.
2Reliability
If the timeout period is extended to prevent split brain, then system reliability improves, but the speed of failure detection and failover response deteriorates
Solution Approach 1:
The timeout period dynamically adapts based on actual network conditions rather than using a single fixed value. During normal operation with stable heartbeat reception, the timeout remains short enabling fast failover detection. When network instability is detected through missed heartbeats, the timeout extends automatically to prevent false positives, thus maintaining both speed and accuracy in different operational states.
Solution Approach 2:
The system changes the timeout parameter based on the reception state of heartbeat packets. Instead of using a static timeout value, the system adjusts this critical parameter dynamically - keeping it short when heartbeats are received normally for fast response, and extending it when packet loss or delay occurs to maintain detection accuracy.
3Measurement precision
If heartbeat packets are transmitted frequently to ensure reliable monitoring, then failure detection precision improves, but the energy consumption and network load increase
Solution Approach 1:
Instead of continuously transmitting heartbeat packets at high frequency regardless of conditions, the system uses partial action by adjusting the monitoring intensity based on network state. When network conditions are stable, normal-frequency heartbeats suffice. When instability is detected, the system temporarily increases monitoring frequency only during the problematic period, reducing overall energy consumption while maintaining detection precision when needed.
Data Source
AI summary
In a cluster system having two server apparatuses that operate as an in-use type or a standby type, a first server apparatus is provided with: a heartbeat transmission/reception part that transmits and receives heartbeat packets to and from a corresponding second server apparatus; and a corresponding node monitoring part that adjusts a timeout period for transitioning operation of the first server apparatus from a standby type to an in-use type, according to the reception state of the heartbeat packets.


