Storage Resync Transfer with Background Mapping Pulls
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The rebuilding of a data warehouse at a storage site after a failure requires significant time, exceeding disaster recovery service level objectives, due to the need to rebuild the entire warehouse even for incremental data modifications, leading to prolonged resync times.
Innovation Solution
The technique involves logging virtual volume block number mappings into a log metafile in parallel with data modifications during the resync process, allowing for immediate data transfer without waiting for the warehouse rebuild, and using a background scanner to asynchronously process these mappings to rebuild the warehouse efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the data warehouse is rebuilt sequentially before data transfer during resync operation, then the warehouse rebuild is completed, but the resync time exceeds disaster recovery service level objectives
Solution Approach 1:
The patent performs preliminary actions by logging virtual volume block number mappings into a log metafile during the resync operation before the data warehouse rebuild is complete. This allows the system to prepare mapping information in advance, so that when the rebuild completes, the data transfer can immediately utilize these pre-computed mappings without waiting for the sequential rebuild to finish first.
Solution Approach 2:
The patent enables continuous useful action by allowing the data transfer process to proceed concurrently with the warehouse rebuild operation. Instead of pausing data transfer to complete the rebuild, the system maintains continuous data synchronization by using the log metafile to track and apply mappings as they become available, eliminating idle time and keeping the resync process continuously productive.
2Reliability
If the entire data warehouse is rebuilt even for incremental data modifications, then data consistency is ensured, but the rebuild time and resources are excessively consumed
Solution Approach 1:
The patent extracts only the essential mapping information (virtual volume block number mappings) from the complete data warehouse rebuild process and stores it in a log metafile. This extraction allows the system to separate the critical mapping data from the full warehouse structure, enabling data transfer to proceed using only the necessary mapping information without requiring the entire warehouse to be fully rebuilt first.
Solution Approach 2:
The patent applies partial action by performing only the necessary mapping logging operation during resync rather than completing the full warehouse rebuild. The system does what is minimally required (logging mappings to metafile) to enable data transfer, avoiding the excessive action of completing the entire sequential rebuild process before allowing any data synchronization to occur.
Data Source
AI summary
Techniques are provided for performing a resync transfer to recover from a storage site failure. During normal operation of a first site hosting a first volume, data is replicated to a second volume hosted by a second site. If the first site fails, when clients are redirected to the second volume at the second site. When the first site recovers, data modifications made to the second volume are resynced back to the first volume. As part of synchronizing the first volume, a data warehouse is rebuilt at the first site in order to track the location of blocks present on the replication destination. Typically, the data modifications are transferred after the data warehouse is rebuilt, which results in significantly long resync times. The techniques provided herein decrease the resync time by either rebuilding the data warehouse in parallel with resyncing the data modifications or circumvent the need for rebuild.