Cross-Site Storage Failover Configuration Re-Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cross-site high-availability storage solutions face challenges in maintaining zero recovery point objective (RPO) and zero recovery time objective (RTO) protections after a failover, due to inconsistencies in data replication configurations and roles between primary and secondary data serving entities, leading to potential split-brain scenarios and disruptions.
Innovation Solution
The solution involves re-aligning the data replication configuration by determining the true primary entity based on consensus to serve I/O, and performing atomic database transactions to update the configuration databases, ensuring that one entity is consistently configured as the source and the other as the destination of data replication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data replication configuration is not re-aligned after failover, then system availability is maintained, but data integrity and zero RPO protection are lost
Solution Approach 1:
The system automatically detects configuration inconsistencies after failover and triggers re-alignment operations without human intervention. The configuration management system monitors itself and performs corrective actions, eliminating the need for manual configuration tracking and ensuring data integrity is maintained autonomously.
Solution Approach 2:
The system implements continuous monitoring of replication configuration states and provides feedback loops that detect inconsistencies between primary and secondary entities. When mismatches are detected, the system automatically initiates re-alignment operations to restore proper configuration, ensuring ongoing data integrity through active feedback control.
2Reliability
If atomic database transactions are used to update configuration, then configuration consistency is ensured, but transaction processing time increases
Solution Approach 1:
The system performs configuration validation and preparation steps before executing atomic transactions. By pre-checking configuration states and preparing transaction payloads in advance, the system reduces the actual transaction execution time while maintaining atomicity guarantees, thus balancing consistency requirements with time efficiency.
3Reliability
If split-brain scenarios are prevented through configuration re-alignment, then data integrity is maintained, but system complexity increases
Solution Approach 1:
The system introduces a configuration management intermediary layer that mediates between primary and secondary entities during failover events. This intermediary coordinates configuration updates, ensures mutual exclusion of conflicting operations, and prevents split-brain scenarios by acting as a central authority that manages the transition state between failover events.
Data Source
AI summary
Systems and methods for re-aligning data replication configuration of a cross-site storage solution after a failover are provided. According to one embodiment, after a failover, the new primary distributed storage system orchestrates flipping of the data replication configuration of a peered consistency group (CG) to reestablish zero RPO and zero RTO protections for the peered CG. The primary causes the secondary distributed storage system to perform an atomic database operation on its remote configuration database to (i) delete an existing source configuration that identifies the secondary as a source of data replication; and (ii) persist a new destination configuration identifying the secondary as a destination of data replication. Then, the primary performs an atomic database operation on its local configuration database to (i) delete an existing destination configuration identifying the primary as the destination; and (ii) persist a new source configuration identifying the distributed storage system as the source.