Virtual Index Deduplication for Asynchronous Storage Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional deduplication techniques for asynchronous replication consume significant computing resources and bandwidth due to the reliance on strong hash signatures, and advanced techniques using similarity hash signatures do not guarantee identical data matches, making efficient data transfer impractical.
Innovation Solution
The method employs a virtual index to identify unique data on the source system and a deduplication index table to track data transferred to the target system, allowing only new data to be sent during replication, reducing bandwidth consumption and storage needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If strong hash signatures are used for deduplication in asynchronous replication, then data integrity and accuracy are improved, but computing resources and bandwidth consumption increase significantly
Solution Approach 1:
The patent segments the data identification process into multiple stages: first using a fast similarity hash signature for initial matching, then using a stronger hash signature only for verification of potential matches. This segmentation allows the system to maintain data integrity through strong hashing while reducing overall computing resource consumption by limiting its application to only necessary cases.
Solution Approach 2:
The patent applies different quality levels of hash signatures to different data scenarios. For most data comparisons, a lighter similarity hash signature is used providing sufficient quality for identification. For critical verification steps, a stronger hash signature is applied locally to ensure data integrity, thus optimizing the balance between reliability and resource consumption.
2Reliability
If strong hash signatures are used for deduplication in asynchronous replication, then data integrity is improved, but bandwidth consumption increases significantly
Solution Approach 1:
The patent segments the data transmission process by first comparing similarity hash signatures to identify potential duplicates locally, then only transmitting data for cases where no match is found or verification is needed. This segmentation reduces bandwidth consumption by avoiding unnecessary transmission of duplicate data while maintaining data integrity through selective use of strong hashing.
Solution Approach 2:
The patent performs preliminary action by comparing similarity hash signatures before actual data transmission. This preliminary filtering step identifies duplicate data that can be skipped, reducing the amount of data that needs to be transmitted over the network while preserving data integrity for non-duplicate data through subsequent strong hash verification.
3Loss of energy
If similarity hash signatures are used for deduplication, then bandwidth consumption is reduced, but data match accuracy is not guaranteed
Solution Approach 1:
The patent segments the deduplication verification process into two stages: first using similarity hash signatures for rapid initial matching to identify potential duplicates, then using strong hash signatures for verification of these candidates. This segmentation maintains low bandwidth consumption by using similarity hashing for filtering while ensuring data match accuracy through the verification stage.
Solution Approach 2:
The patent uses similarity hash signatures as an intermediary step between data comparison and final verification. This intermediary filtering mechanism reduces the number of data blocks requiring full verification, thereby reducing bandwidth consumption while the subsequent strong hash verification ensures that data match accuracy is maintained for the filtered set.
4Reliability
If all data is transferred during asynchronous replication, then data completeness is improved, but bandwidth consumption and storage needs increase
Solution Approach 1:
The patent performs preliminary action by comparing similarity hash signatures of source data with target data before initiating full data transfer. This preliminary step identifies duplicate data blocks that can be skipped, ensuring data completeness for non-duplicate blocks while significantly reducing bandwidth consumption by avoiding transmission of redundant data.
Solution Approach 2:
The patent applies discarding and recovering by identifying and discarding duplicate data blocks through similarity hash comparison, then recovering data completeness by transferring only the unique data blocks. This approach maintains data completeness for the replicated system while minimizing bandwidth consumption by discarding redundant transmissions.
Data Source
AI summary
Techniques are used for enabling deduplication for asynchronous replication in a storage system. The techniques may be used to provide, among other things, retrieving, on a source system, a virtual index key corresponding to a logical address, and determining whether a deduplication index table for a target system includes an entry with the same virtual index key.