Failback Resync Transfer Using Background Pull and Log Metafiles
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The rebuilding of the entire data warehouse during a resync operation after a storage site failure consumes significant time, exceeding disaster recovery service level objectives and impacting other processes like asynchronous and synchronous replication, due to the need to rebuild the warehouse for the entire volume rather than just the data modifications.
Innovation Solution
A log metafile is used to log virtual volume block number mappings in parallel with data modifications during the resync transfer, allowing for asynchronous processing by a background scanner to reduce the time required for the resync operation by eliminating the need to rebuild the entire data warehouse before transferring data modifications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire data warehouse is rebuilt during resync operation, then data consistency is ensured, but the resync transfer time exceeds disaster recovery service level objectives
Solution Approach 1:
The patent segments the data warehouse rebuild process into two independent parts: (1) logging virtual volume block number mappings to a log metafile during data modifications, and (2) asynchronous processing by a background scanner to populate the data warehouse. This allows the resync operation to proceed without waiting for the complete warehouse rebuild, thus reducing transfer time while maintaining data consistency.
Solution Approach 2:
The patent performs preliminary actions by logging virtual volume block number mappings to a log metafile during the resync operation itself, rather than waiting until after the resync completes. This preliminary logging enables the background scanner to process and populate the data warehouse asynchronously, eliminating the bottleneck that previously caused extended resync transfer times.
2Reliability
If the entire data warehouse is rebuilt before data modification transfer, then data integrity is maintained, but other processes like asynchronous and synchronous replication are impacted
Solution Approach 1:
The patent separates the data warehouse population process from the data modification transfer process. The log metafile captures virtual volume block number mappings during resync, and the background scanner independently processes this log to populate the data warehouse. This segmentation allows replication processes to continue without being blocked by the warehouse rebuild, maintaining both data integrity and replication efficiency.
Solution Approach 2:
The log metafile serves as an intermediary mechanism that decouples the resync operation from the data warehouse population process. By logging mappings during resync and having the background scanner process this log asynchronously, the system maintains data integrity while allowing other replication processes to operate independently and efficiently.
Data Source
AI summary
Techniques are provided for performing a resync transfer to recover from a storage site failure. During normal operation of a first site hosting a first volume, data is replicated to a second volume hosted by a second site. If the first site fails, when clients are redirected to the second volume at the second site. When the first site recovers, data modifications made to the second volume are resynced back to the first volume. As part of synchronizing the first volume, a data warehouse is rebuilt at the first site in order to track the location of blocks present on the replication destination. Typically, the data modifications are transferred after the data warehouse is rebuilt, which results in significantly long resync times. The techniques provided herein decrease the resync time by either rebuilding the data warehouse in parallel with resyncing the data modifications or circumvent the need for rebuild.