Storage Data Migration Deduplication and Delta Transfer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data migration technologies face inefficiencies due to narrow connections between storage devices, leading to slow migration speeds and increased costs when trying to enhance traffic performance by installing new switch devices.
Innovation Solution
The technique offloads traffic between storage devices to internal connections by deduplicating data before migration, transferring only deduplicated and difference data once, and then expanding deduplication on the target storage device, allowing for simultaneous migration of volumes and consistency groups with the same configuration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data migration is performed using narrow connections between storage devices, then migration can be implemented with existing infrastructure, but migration speed becomes slow
Solution Approach 1:
The system performs deduplication of data before migration by comparing source and target volumes in advance. By identifying and eliminating duplicate data blocks beforehand, the system reduces the amount of data that needs to be transferred over the narrow connection, thereby increasing migration speed without requiring bandwidth expansion
Solution Approach 2:
The system extracts only the difference data (delta) between source and target volumes for migration, rather than transferring the entire dataset. This selective migration approach minimizes the data volume over the narrow connection, resolving the contradiction between migration speed and connection bandwidth requirements
2Productivity
If new switch devices are installed to increase traffic performance, then migration speed improves, but cost increases
Solution Approach 1:
The system creates a copy of the data structure (delta bitmap) that maps difference data locations before actual migration. This copy mechanism enables the system to identify and transfer only necessary data blocks, achieving high migration speed using existing infrastructure without requiring expensive switch upgrades
Solution Approach 2:
By performing deduplication analysis in advance and creating delta bitmaps that record only difference data locations, the system prepares migration plans that minimize data transfer requirements. This preliminary preparation enables fast migration using existing network infrastructure, avoiding the need for costly switch device installations
3Productivity
If all volumes are migrated simultaneously to reduce migration time, then productivity improves, but traffic load on connections increases
Solution Approach 1:
The system extracts and transfers only difference data (delta) between volumes rather than all data. By using delta bitmaps to identify and transfer only the minimal necessary data blocks, the system achieves high throughput during simultaneous migration while keeping actual data transfer volume low, thus avoiding connection overload
Solution Approach 2:
The system performs preliminary deduplication analysis and creates delta bitmaps that record only difference data locations before migration begins. This advance preparation enables the system to migrate all volumes simultaneously while transferring minimal data, as the delta bitmaps guide selective transfer of only necessary blocks, preventing traffic overload
Data Source
AI summary
Exemplary embodiments provide a technique to reduce the traffic between storage devices during data migration. In one embodiment, a system comprises a plurality of storage systems which are operable to migrate a set of primary and secondary volumes between the storage systems by managing and copying, between the storage systems, a plurality of same data and a plurality of difference data between the primary and secondary volumes and location information of each of the plurality of difference data, the location information identifying a location of the difference data in the primary or secondary volume associated with the difference data. Each secondary volume which corresponds to a primary volume, if said each source secondary volume contains data, has a same data as the primary volume and, if said each secondary volume is not synchronized with the primary volume, further has a difference data with respect to the primary volume.