Deduplication-Aware Replication Using Unreliable Hash Signatures
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data deduplication in address-based storage systems is costly in terms of CPU and memory, and existing methods do not efficiently utilize hash signatures for deduplication, especially in long-distance data replication where bandwidth costs are high.
Innovation Solution
A method that uses unreliable and reliable hash signatures to determine if data pages exist on a target, allowing for data-less write commands to deduplicate data, with the option to replicate using both hash signatures when deduplication thresholds are met, thereby minimizing data transmission.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If hash signatures are kept for every page to enable deduplication, then data deduplication capability is improved, but CPU and memory costs increase significantly
Solution Approach 1:
The patent segments the hash signature verification process into two stages: first using an unreliable but fast hash function to quickly identify potential duplicates, then using a reliable but computationally expensive hash function only when needed. This segmentation reduces the overall CPU and memory cost while maintaining deduplication capability.
Solution Approach 2:
The patent employs inexpensive, computationally light unreliable hash signatures as the primary filtering mechanism. These cheap hash computations are performed on every page to identify potential duplicates, sacrificing some accuracy for significant computational savings, with expensive reliable hash verification used only as a fallback.
2Reliability
If data is replicated from source to target, then data availability is improved, but bandwidth costs increase significantly
Solution Approach 1:
The patent performs preliminary deduplication checks using hash signatures before initiating data replication. By pre-identifying duplicate pages through hash comparison, the system avoids unnecessary data transmission, significantly reducing bandwidth costs while ensuring data availability through selective replication.
Solution Approach 2:
The patent extracts only the unique, non-duplicate pages for replication while leaving duplicate pages on the target. This extraction approach ensures data availability for unique content while eliminating wasted bandwidth on redundant data transmission.
3Measurement precision
If reliable hash verification is performed on every page, then deduplication accuracy is improved, but processing time increases
Solution Approach 1:
The patent divides the verification process into two segments: an initial fast filtering stage using unreliable hash comparison, and a secondary accurate verification stage using reliable hash functions only for pages that pass the initial filter. This segmentation maintains high deduplication accuracy while minimizing processing time.
Solution Approach 2:
The patent applies partial reliable hash verification only to the subset of pages that fail the unreliable hash check, rather than verifying every page with the expensive reliable method. This partial action approach maintains accuracy for critical cases while dramatically reducing overall processing time.
Data Source
AI summary
A method, computer program product, and computer system for receiving, by a target sent from a source, a first hash signature associated with a page of data. It may be determined that the first hash signature exists on the target. The target may receive a second hash signature sent from the source associated with the page of data. A third hash signature may be generated at the target. It may be determined that the second hash signature matches the third hash signature indicating the page of data exists on the target. A data-less write command may be executed using the page of data existing on the target to deduplicate the page of data existing on the target.