Deduplication Digest Bucketing for Partial Block Matches
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional deduplication schemes require exact matches between hash values, missing opportunities for data reduction due to large differences in hash values even with small differences in blocks, such as varying timestamps or metadata.
Innovation Solution
An improved technique assigns digest values to buckets in a deduplication database, allowing for the identification of both exact and partial matches by generating similarity hash values and performing clustering on sim hashes, enabling deduplication even when blocks contain differing portions of data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If conventional exact-match deduplication schemes are used, then hash value matching is simple and fast, but data reduction opportunities are lost when blocks have minor differences
Solution Approach 1:
The digest value is segmented into multiple portions, with each portion assigned to a separate bucket. This allows the system to search for partial matches by examining individual bucket portions independently, enabling data reduction for blocks with minor differences while maintaining manageable complexity through localized search operations.
Solution Approach 2:
The patent introduces a new dimension to hash matching by dividing the digest into multiple bucket portions and allowing partial matches across these portions. This transforms the traditional single-dimension exact-match approach into a multi-dimensional partial-match system, enabling data reduction for similar blocks without requiring complete hash value matches.
2Loss of substance
If similarity hash values and clustering are used to find partial matches, then data reduction is improved, but search complexity and processing time increase
Solution Approach 1:
By segmenting the digest into multiple bucket portions and organizing them in a hierarchical structure, the system enables efficient partial-match searches. The segmentation allows the search to proceed through discrete portions rather than requiring exhaustive comparison of entire hash values, reducing search time while maintaining improved data reduction capabilities.
Solution Approach 2:
The digest value is pre-divided into multiple bucket portions and organized in advance before the deduplication search begins. This preliminary organization of hash portions enables faster search operations by allowing the system to quickly navigate to relevant buckets and compare only the necessary portions, rather than performing time-consuming full hash comparisons during the deduplication process.
3Measurement precision
If all buckets are searched for partial matches, then match accuracy is maximized, but processing overhead increases significantly
Solution Approach 1:
The segmentation of digest into multiple bucket portions allows the system to search through organized segments rather than examining all buckets indiscriminately. This segmentation maintains match accuracy by preserving the ability to identify partial matches while improving processing throughput through structured navigation and reduced search scope.
Solution Approach 2:
Different bucket portions are treated with different search strategies based on their local characteristics. The system can focus search efforts on buckets with higher likelihood of containing matches, applying varying levels of search intensity to different portions. This local quality approach maintains overall match accuracy while optimizing processing throughput by avoiding exhaustive search of all buckets.
Data Source
AI summary
A technique for performing deduplication traverses a deduplication database and assigns digest values in the database to buckets, where each bucket covers a respective range of digest values. To deduplicate a particular candidate block, the technique generates a digest from the candidate block and searches for the computed digest in a subset of the buckets, where the subset is selected based on the computed digest. If a target block providing an exact match or a suitably close partial match is found in the subset of buckets, the technique effects storage of the candidate block at least in part by providing a reference to the target block.