Resync Transfer Using Log Metafile for Storage Site Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing data replication and resync processes in storage environments are inefficient, particularly during site failures and recovery, as they require rebuilding the entire data warehouse, leading to prolonged resync times that exceed disaster recovery service level objectives.
Innovation Solution
The proposed solution involves logging virtual volume block number mappings into a log metafile during the resync operation, allowing for parallel processing of data modifications and block sharing, thereby reducing the need to rebuild the entire data warehouse.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire data warehouse is rebuilt during resync operation, then data consistency between sites is ensured, but the resync time becomes excessively long (hours or days)
Solution Approach 1:
The patent extracts only the necessary mapping information (virtual volume block number mappings) from the data warehouse and stores it separately in a log metafile. During resync, only this extracted mapping data needs to be transferred and processed, rather than rebuilding the entire data warehouse structure, thus dramatically reducing resync time while maintaining data consistency
Solution Approach 2:
The patent performs preliminary logging of virtual volume block number mappings to the log metafile during normal operation before failure occurs. This preliminary action ensures that when resync is needed, the mapping information is already prepared and available, eliminating the need to reconstruct it during the critical resync operation
2Loss of energy
If sequential processing of data warehouse rebuild is used, then system resources are conserved, but productivity during resync operation is reduced
Solution Approach 1:
The patent segments the resync operation into independent parallel tasks: multiple workers can simultaneously transfer different portions of the log metafile, process different mapping entries, and update different data blocks. This segmentation enables parallel processing that increases productivity while keeping individual worker resource consumption manageable
Solution Approach 2:
The patent implements partial processing by allowing the resync operation to process only the necessary mapping entries from the log metafile rather than the entire data warehouse. Multiple workers can perform partial actions concurrently, with each worker handling a subset of the total work, thus achieving high throughput without requiring excessive resources per worker
Data Source
AI summary
Techniques are provided for performing a resync transfer to recover from a storage site failure. During normal operation of a first site hosting a first volume, data is replicated to a second volume hosted by a second site. If the first site fails, when clients are redirected to the second volume at the second site. When the first site recovers, data modifications made to the second volume are resynced back to the first volume. As part of synchronizing the first volume, a data warehouse is rebuilt at the first site in order to track the location of blocks present on the replication destination. Typically, the data modifications are transferred after the data warehouse is rebuilt, which results in significantly long resync times. The techniques provided herein decrease the resync time by either rebuilding the data warehouse in parallel with resyncing the data modifications or circumvent the need for rebuild.