Split-Brain Detection in Redundant Processing Units
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High availability systems face the 'split-brain' condition where both processors attempt to provide service due to false detection of failure, leading to service disruption, as they cannot differentiate between actual unit failure and communication channel failure.
Innovation Solution
Implementing redundant links, a backchannel connection, and power cycler switches between processors to diagnose and correct communication failures, and using keepalive signals and state machines to differentiate between unit and interconnect failures, allowing for timely and accurate corrective actions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If heartbeat signals are sent over the interconnect to detect peer failure, then failure detection speed is improved, but false detection of failure occurs when the interconnect itself fails
Solution Approach 1:
The system segments the communication path by introducing a backchannel as a separate communication path from the interconnect. This allows independent verification of peer unit status, preventing false failure detection when the interconnect fails. The backchannel and interconnect operate as separate segments that can be independently monitored.
Solution Approach 2:
The backchannel acts as an intermediary communication path that mediates between the peer units for failure detection purposes. By using this intermediate channel separately from the primary interconnect, the system can verify whether a peer unit is truly failed or if the interconnect itself is the problem.
2Reliability
If the Standby processor takes over when peer failure is detected, then service continuity is improved, but service disruption occurs when both processors attempt to provide service due to split-brain condition
Solution Approach 1:
The system implements feedback mechanisms where each processor sends and receives keepalive signals over both the interconnect and backchannel. This continuous feedback loop allows processors to accurately determine peer status and coordinate failover actions, preventing the split-brain condition where both processors simultaneously assume active service.
Solution Approach 2:
The system performs preliminary diagnostic actions before initiating failover. By checking both interconnect and backchannel communication status, and using state machines to evaluate the situation, the system determines the true cause of communication loss before allowing Standby to take over, preventing premature or incorrect failover decisions.
3Measurement precision
If diagnostic checks are performed to differentiate between unit failure and interconnect failure, then diagnosis accuracy is improved, but response time is increased
Solution Approach 1:
The system uses periodic keepalive signals sent at regular intervals over both interconnect and backchannel to continuously monitor peer status. This periodic action provides timely detection of failures without requiring complex on-demand diagnostic sequences, balancing diagnosis speed with accuracy.
Solution Approach 2:
The state machines are pre-configured with diagnostic logic that automatically executes upon detecting communication anomalies. This preliminary preparation of diagnostic routines enables rapid automated differentiation between unit failures and interconnect failures without requiring time-consuming manual analysis or complex sequential testing.
Data Source
AI summary
In an example embodiment the occurrence of the split-brain condition in a High-Availability system, having active and standby processing units, is detected, its cause is diagnosed, and the cause is treated to prevent interruption of service. Diagnosis and treatment procedures are performed at the active processing unit prior to being performed at the standby processing unit.