Sub-block Hash Distance for Data Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional deduplication schemes fail to detect redundant data when blocks are similar but not identical, leading to missed opportunities for improving storage efficiency due to the randomizing nature of hash functions.
Innovation Solution
The technique operates at sub-block granularity by searching a deduplication database for matches between candidate and target sub-blocks, identifying duplicate ranges, and configuring mapping metadata to store only unique portions of data blocks, thereby avoiding redundant storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If conventional hash-based deduplication is used, then deduplication speed is improved, but detection accuracy of similar blocks deteriorates
Solution Approach 1:
The patent divides data blocks into sub-blocks and computes hashes for each sub-block individually. This segmentation allows the system to detect partial matches within blocks, improving detection accuracy while maintaining the speed benefits of hash-based comparison. The sub-block hashing approach enables identification of similar regions even when entire blocks differ.
Solution Approach 2:
The patent applies different processing approaches to different parts of the data structure. Specifically, it uses hash comparison for quick filtering at the block level, then applies more detailed sub-block analysis only where needed. This local differentiation optimizes both speed and accuracy by avoiding unnecessary detailed comparisons while ensuring thorough detection where similarity exists.
2Loss of substance
If sub-block granularity deduplication is implemented, then storage efficiency is improved, but computational complexity increases
Solution Approach 1:
The patent performs preliminary hash computation on sub-blocks before actual storage operations. By pre-computing and storing sub-block hashes, the system establishes an index structure that enables rapid detection of duplicates during write operations. This preliminary action reduces the computational burden during actual deduplication processing.
Solution Approach 2:
The patent introduces sub-block hashes as an intermediary data structure between raw data and storage decisions. These hashes serve as mediators that enable efficient comparison without requiring direct byte-by-byte analysis of entire blocks. The intermediary hash structure simplifies the computational complexity while maintaining accurate duplicate detection.
3Measurement precision
If distance function based similarity measurement is used, then detection accuracy is improved, but processing time increases
Solution Approach 1:
The patent applies distance function-based similarity measurement selectively rather than universally. It uses sub-block hash comparison as a fast preliminary filter, and only applies the more computationally intensive distance function when hashes indicate potential similarity. This partial application of the more accurate method maintains detection accuracy while minimizing processing time overhead.
Data Source
AI summary
Techniques for processing data may include: receiving a candidate data block; computing a distance using a distance function, wherein the distance denotes a measurement of similarity between the candidate data block and a target data block; and determining, using the distance, whether to perform data deduplication of the candidate data block with respect to the target data block to identify at least one sub-block of the candidate data block that is a duplicate of at least one sub-block of the target data block. The distance may be computed using a bit-wise logical exclusive-or operation of the contents of the candidate data block and the target data block. The distance may be computed using a bit-wise logical exclusive-or operation of digests computed for the candidate and target data blocks using a distance preserving hash function. The target and candidate block may be similar if the distance is less than a threshold.