Synchronous Replication Automatic Recovery State Machine
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Synchronous replication systems face challenges in resuming and recovering from out-of-sync conditions after a source site disaster, requiring time-consuming and bandwidth-intensive full resyncs due to the lack of incremental recovery mechanisms.
Innovation Solution
A state machine-based method that transitions between sync and async states to facilitate automatic recovery, monitoring IO operations and initiating remote data transfer operations to return the system to a synchronous state, allowing for incremental recovery and reducing manual intervention.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If synchronous replication is used to ensure data consistency, then data reliability is improved, but recovery time after failover increases due to the need for full re-sync
Solution Approach 1:
The system performs preliminary actions by maintaining a list of synchronized snap sets during normal operation, which enables incremental recovery after failover. This preliminary tracking mechanism allows the system to resume replication from the last synchronized point rather than requiring a full re-sync, thus reducing recovery time while maintaining data consistency.
Solution Approach 2:
The system changes the replication mode parameter from synchronous to asynchronous during the recovery process. By transitioning to async mode, the system can perform incremental recovery using the tracked snap sets, and then transition back to sync mode once recovery is complete. This parameter change enables faster recovery without sacrificing the data consistency guarantees of synchronous replication.
2Reliability
If synchronous replication is used to prevent data loss, then data safety is improved, but bandwidth consumption increases during recovery operations
Solution Approach 1:
The system segments the recovery process into two phases: first, an incremental recovery phase using tracked snap sets to transfer only unsynced data; second, a final sync phase to ensure data consistency. This segmentation reduces bandwidth consumption during recovery by avoiding unnecessary transfer of already-synchronized data, while still ensuring data safety through the final synchronization check.
Solution Approach 2:
The system changes the replication parameter from synchronous to asynchronous during the recovery phase, allowing incremental data transfer based on tracked snap sets. This parameter change enables the system to transfer only the necessary data portions, significantly reducing bandwidth consumption compared to a full synchronous re-sync, while maintaining data safety through proper validation and state tracking.
3Device complexity
If manual intervention is required for recovery operations, then system complexity is reduced, but operational efficiency decreases
Solution Approach 1:
The system implements self-service by automatically tracking synchronized snap sets during normal operation and autonomously executing the recovery process after failover. The state machine automatically transitions between sync and async modes, manages the incremental recovery process, and restores synchronous replication without requiring manual intervention. This automation improves operational efficiency while the structured approach keeps system complexity manageable.
Solution Approach 2:
The system uses feedback mechanisms through the state machine that continuously monitors replication status and snap set synchronization. This feedback enables the system to automatically detect when recovery is complete and transition back to synchronous mode, ensuring operational efficiency through automated decision-making while maintaining clear system state management to control complexity.
Data Source
AI summary
In one aspect, automatic recovery of a synchronous replication session in response to an error is provided for a storage system that includes a source and target sites. During an active sync replication session in which a state machine indicates the system is operating in sync, an aspect includes monitoring input/output (IO) operations. Upon determining an occurrence of the error in which data has been persisted at the source site but not at the target site, an aspect includes discontinuing replication to the target site and transitioning the state machine from a sync state to a tripped state. Upon determining, during the tripped state, resources exist to conduct sync replication remote data transfer operations, transition the state machine to an async_to_sync state. The async_to_sync state causes the storage system to initiate a recovery operation to return the source and target sites to the sync state.