Dynamic Heartbeat Timeout for Cluster Split Brain Prevention

Resolve Bottlenecks,
Find Innovative Solutions
Generate Solutions

Solution Overview

Problem

In ACT-SBY cluster configurations, the occurrence of split brain due to communication failures in heartbeat packets leads to inconsistent data and service disruptions, making it challenging to maintain high reliability and availability, especially in carrier-oriented systems.

Innovation Solution

A server apparatus with a heartbeat transmission/reception part and a corresponding node monitoring part that dynamically adjusts the timeout period based on the reception state of heartbeat packets, preventing erroneous failover and maintaining service integrity by ensuring accurate node failure detection and minimizing service impact.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If a fixed timeout period is used for heartbeat packet monitoring, then the system structure is simple and easy to implement, but the system reliability deteriorates due to split brain occurrences during network communication failures

Engineering Contradiction:
Improvesystem reliabilityVSAvoidmonitoring mechanism complexity
Core Design Contradiction:
ReliabilityVSDevice complexity

Solution Approach 1:

The timeout period is changed from a fixed value to a dynamic value that adjusts based on the reception state of heartbeat packets. When heartbeat packets are received normally, the timeout period remains at a standard value. When packet loss or delay is detected, the timeout period is extended automatically. This dynamic adjustment prevents erroneous failover during network issues while maintaining simple operation during normal conditions.

Inventive Principle:
Principle #15Dynamics

Solution Approach 2:

The monitoring mechanism incorporates feedback by continuously observing the reception state of heartbeat packets and using this information to adjust the timeout period. The system monitors whether heartbeat packets are received within expected timeframes and modifies the timeout threshold accordingly, creating a closed-loop control system that adapts to network conditions.

Inventive Principle:
Principle #23Feedback

2Reliability

If the timeout period is extended to prevent split brain, then system reliability improves, but the speed of failure detection and failover response deteriorates

Engineering Contradiction:
Improvefailure detection accuracyVSAvoidfailover response speed
Core Design Contradiction:
ReliabilityVSSpeed

Solution Approach 1:

The timeout period dynamically adapts based on actual network conditions rather than using a single fixed value. During normal operation with stable heartbeat reception, the timeout remains short enabling fast failover detection. When network instability is detected through missed heartbeats, the timeout extends automatically to prevent false positives, thus maintaining both speed and accuracy in different operational states.

Inventive Principle:
Principle #15Dynamics

Solution Approach 2:

The system changes the timeout parameter based on the reception state of heartbeat packets. Instead of using a static timeout value, the system adjusts this critical parameter dynamically - keeping it short when heartbeats are received normally for fast response, and extending it when packet loss or delay occurs to maintain detection accuracy.

Inventive Principle:
Principle #35Parameter changes

3Measurement precision

If heartbeat packets are transmitted frequently to ensure reliable monitoring, then failure detection precision improves, but the energy consumption and network load increase

Engineering Contradiction:
Improvefailure detection precisionVSAvoidenergy consumption
Core Design Contradiction:
Measurement precisionVSUse of energy by moving object

Solution Approach 1:

Instead of continuously transmitting heartbeat packets at high frequency regardless of conditions, the system uses partial action by adjusting the monitoring intensity based on network state. When network conditions are stable, normal-frequency heartbeats suffice. When instability is detected, the system temporarily increases monitoring frequency only during the problematic period, reducing overall energy consumption while maintaining detection precision when needed.

Inventive Principle:
Principle #16Partial or excessive action

Data Source

PatentUS10911295B2Server apparatus, cluster system, cluster control method and program
Publication Date: 2021.02.02 NEC CORP
  • US10911295B2 patent drawing
  • US10911295B2 patent drawing
  • US10911295B2 patent drawing

AI summary

In a cluster system having two server apparatuses that operate as an in-use type or a standby type, a first server apparatus is provided with: a heartbeat transmission/reception part that transmits and receives heartbeat packets to and from a corresponding second server apparatus; and a corresponding node monitoring part that adjusts a timeout period for transitioning operation of the first server apparatus from a standby type to an in-use type, according to the reception state of the heartbeat packets.