Storage Resynchronization via Pending IO Request Marking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data storage systems face challenges in efficiently resynchronizing storage resources across sites after a network failure, particularly due to the reliance on periodic recovery snapshots that may be outdated, leading to increased network transfer and longer recovery times.
Innovation Solution
The system employs pending IO requests to identify data differences between source and destination storage systems, creating snapshots and marking blocks as modified to facilitate remote mirroring of differences once the network connection is restored, thereby reducing the need for large-scale network transfers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If periodic recovery snapshots are used for resynchronization after network failure, then data replication can be performed, but the snapshots become outdated and require larger network transfers and longer recovery times
Solution Approach 1:
The system performs preliminary actions by quiescing IO activity and creating snapshots at the moment of network failure detection, rather than relying on periodic snapshots. This preliminary capture of the failure state enables immediate resynchronization when the network is restored, eliminating the delay of waiting for the next periodic snapshot cycle.
Solution Approach 2:
The system dynamically adapts the resynchronization process by detecting network failure conditions and automatically adjusting from periodic snapshot-based replication to an event-driven approach using pending IO requests. This dynamic adaptation allows the system to respond to actual failure conditions rather than following a fixed periodic schedule, reducing recovery time.
2Reliability
If periodic recovery snapshots are used for resynchronization, then data can be replicated, but network transfer volume increases due to outdated snapshot data
Solution Approach 1:
The system extracts only the necessary data for resynchronization by using pending IO requests to identify exactly which blocks need to be transferred. Instead of transferring all data from an outdated periodic snapshot, the system extracts and transfers only the specific blocks that were pending at the time of failure, significantly reducing network transfer volume.
Solution Approach 2:
The system changes the parameter of snapshot freshness from periodic intervals to failure-time accuracy. By creating snapshots based on the actual failure event rather than a fixed schedule, the system ensures that the snapshot reflects the true state of data at failure, minimizing the amount of data that needs to be retransferred.
3Productivity
If pending IO requests are used to identify data differences, then resynchronization speed improves, but system complexity increases due to additional snapshot and marking operations
Solution Approach 1:
The system performs preliminary actions by quiescing IO activity and creating snapshots at the moment of network failure detection, rather than relying on periodic snapshots. This preliminary capture of the failure state enables immediate resynchronization when the network is restored, eliminating the delay of waiting for the next periodic snapshot cycle.
Solution Approach 2:
The system uses copying by creating snapshots of storage resources at the failure point and using these snapshots to identify differences. Instead of complex real-time comparison of all data blocks, the system copies the state at failure and uses this copy to efficiently determine what needs to be resynchronized, simplifying the overall process.
4Measurement precision
If IO activity is quiesced to capture pending requests, then accurate data difference identification is achieved, but IO performance deteriorates during the quiescing period
Solution Approach 1:
The system performs preliminary actions by quiescing IO activity and creating snapshots at the moment of network failure detection, rather than relying on periodic snapshots. This preliminary capture of the failure state enables immediate resynchronization when the network is restored, eliminating the delay of waiting for the next periodic snapshot cycle.
Solution Approach 2:
The system dynamically adapts the resynchronization process by detecting network failure conditions and automatically adjusting from periodic snapshot-based replication to an event-driven approach using pending IO requests. This dynamic adaptation allows the system to respond to actual failure conditions rather than following a fixed periodic schedule, reducing recovery time.
Data Source
AI summary
Techniques are disclosed for storage resource resynchronization using pending IO requests. For example, a method includes, upon detecting a failed network: quiescing IO activity in progress; identifying storage resources and associated pending IO requests; for each storage resource: creating a first snapshot of the storage resource; for each pending IO request, marking blocks specified by the pending IO request as modified; unquiescing the quiesced IO activity; upon detecting the network has been restored: creating a second snapshot of each storage resource; remote mirroring, to the destination, a copy of differences of each source storage resource to the destination storage resource, wherein the differences are identified by comparing the first and the second snapshot; transmitting a request for the destination to create a third snapshot comprising a snapshot of each destination storage resource; and triggering the destination to process the third snapshot to identify storage resources based on the differences.