Multi-AZ Arbitration Module for Split-Brain Prevention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The occurrence of split-brain scenarios between availability zones (AZs) in multi-AZ architectures leads to data inconsistency and reduces the reliability of cloud services, as heartbeat packet failures may incorrectly identify AZs as non-functional when they are actually providing services.
Innovation Solution
A multi-AZ arbitration system with an arbitration module that determines network states and primary/secondary node locations based on detection information and user-configured policies, reducing the likelihood of split-brain by intelligently managing service provision across AZs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If heartbeat packet detection is used to monitor AZ status, then service availability is improved, but false positive rates increase leading to split-brain scenarios
Solution Approach 1:
An arbitration module is introduced as an intermediary component between AZs and the service nodes. This module receives heartbeat packets from all AZs, verifies their authenticity, and makes authoritative decisions about which AZ should provide the service. The arbitration module prevents split-brain scenarios by mediating the communication and coordination between multiple AZs, ensuring that only one AZ is promoted to primary status even when network partitions occur.
Solution Approach 2:
The system implements a feedback mechanism where detection modules continuously monitor network connectivity and heartbeat packets from all AZs, sending this information to the arbitration module. The arbitration module uses this feedback to dynamically determine network states and make real-time decisions about service provision. This feedback loop ensures that the system responds accurately to actual AZ status changes while preventing false promotions based on transient network issues.
2Reliability
If multiple AZs independently provide services, then service redundancy is improved, but data inconsistency occurs due to split-brain
Solution Approach 1:
The arbitration module merges the decision-making capabilities of multiple AZs into a single centralized authority. Instead of each AZ independently deciding whether to provide services, all AZs send their status information to the arbitration module, which consolidates this information and makes unified decisions about service provision. This merging of decision-making functions ensures that only one AZ can actively provide services at a time, preventing data inconsistency while maintaining the redundancy benefits of multiple AZs.
Solution Approach 2:
The system performs preliminary verification of network connectivity and heartbeat packet validity before allowing an AZ to be promoted to primary status. The arbitration module proactively checks the status of all AZs and network connections before making any decisions, preventing split-brain scenarios from occurring in the first place. This preliminary action ensures that only healthy and properly connected AZs are considered for service provision, maintaining data consistency across the system.
3Reliability
If network connectivity between AZs is continuously monitored, then service reliability is improved, but system complexity increases
Solution Approach 1:
The arbitration module is designed as a multi-functional component that handles multiple tasks: receiving heartbeat packets from all AZs, verifying network connectivity, determining network states, making arbitration decisions, and coordinating service provision. By consolidating these multiple functions into a single module, the system achieves high reliability through continuous monitoring while avoiding the complexity that would arise from having separate components for each function. The detection modules also serve multiple purposes by monitoring both connectivity and heartbeat packet validity.
Data Source
AI summary
The arbitration system includes an arbitration module and M AZs. Each AZ includes a detection module and a plurality of service nodes. M is an integer greater than 2. The M AZs are configured to: run at least one application and provide at least one service for each application. The detection module in each AZ is configured to send detection information to the arbitration module. The arbitration module is configured to: receive an arbitration policy configured by a user and detection information from M detection modules; and determine network states between the M AZs based on the detection information of the M detection modules, and determine, based on the network states and the arbitration policy, an AZ in which a primary node that provides each service for an application of the user is located and an AZ in which a secondary node is located.