Automated Storage Failover via Quorum and Heartbeat Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current storage systems require human intervention for failover from a primary to a secondary storage system, leading to disruptive operations due to the inability to automatically determine communication failures from synchronization issues, potentially causing 'split brain' scenarios with inconsistent data.
Innovation Solution
A method for a secondary storage system to autonomously perform failover updates by determining synchronization levels and disconnection status, prioritizing IO requests to itself over the primary system, and reversing this during failback, without human intervention.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If the secondary storage system automatically takes over when disconnected from the primary storage system, then failover speed and automation are improved, but the risk of split brain scenarios and data inconsistency increases
Solution Approach 1:
The system performs preliminary actions by establishing a quorum mechanism before allowing failover. The secondary storage system checks whether it has reached the required quorum threshold (e.g., majority of nodes) before accepting leadership and processing writes, preventing split brain scenarios by ensuring only one system can become leader at a time
Solution Approach 2:
The system implements feedback through continuous heartbeats and quorum status monitoring. The secondary storage system receives feedback about the primary's status and adjusts its behavior accordingly - remaining in standby if primary is healthy, or transitioning to leader if primary fails and quorum is achieved
2Reliability
If the secondary storage system waits for confirmation of primary failure before taking over, then data consistency is maintained, but failover time increases
Solution Approach 1:
The system performs preliminary synchronization of data and metadata to the secondary storage system before failure occurs. This preliminary action ensures that the secondary is ready to take over immediately upon failure detection, minimizing failover time while maintaining consistency
Solution Approach 2:
The system dynamically adjusts the failover trigger conditions based on the situation. Instead of waiting for absolute confirmation of primary failure, the secondary can take over when it detects primary unresponsiveness combined with achieving quorum, balancing between premature failover and excessive waiting
3Reliability
If manual intervention is required for role switching between storage systems, then control over the failover process is improved, but operational disruption and complexity increase
Solution Approach 1:
The storage system performs self-service by automatically detecting primary failure, checking quorum status, and transitioning roles without human intervention. The system manages its own failover process, reducing operational disruption while maintaining controlled and reliable transitions through automated decision-making
Data Source
AI summary
A method for managing a secondary storage system, the method may include performing a failover update, by the secondary storage system, of a status of at least one secondary path that couples a host computer to the secondary storage system thereby causing the host computer to prioritize sending IO requests to the secondary storage system over sending the IO requests to a primary storage system; wherein the failover update is triggered by a determination made by the secondary storage system that (a) the secondary storage system is, when making the determination, at an acceptable synchronization level with the primary storage system, and (b) the primary storage system is disconnected from the host computer; receiving, from the host computer, IO requests; and responding, by the secondary storage system, to the IO requests.