Entropy-Based Sub-Block Deduplication for Similar Data Blocks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional deduplication schemes fail to detect redundant data when blocks are similar but not identical, leading to missed opportunities for improving storage efficiency due to the randomizing nature of hash functions.
Innovation Solution
The technique operates at sub-block granularity by searching a deduplication database for matches between candidate and target sub-blocks, identifying duplicate ranges, and configuring mapping metadata to point to the target block, thereby avoiding redundant storage of identical portions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If conventional hash-based deduplication is used, then deduplication speed is improved, but detection accuracy of similar blocks deteriorates
Solution Approach 1:
The patent divides data blocks into sub-blocks and computes entropy values for each sub-block individually. This segmentation allows the system to identify similar portions within blocks without requiring complete block matches, thereby maintaining speed while improving detection accuracy for partial duplicates.
Solution Approach 2:
The patent transforms the deduplication approach from exact hash matching to entropy-based similarity measurement. By changing the parameter from binary match/no-match to a continuous entropy distance metric, the system can detect similar blocks that would be missed by conventional hashing while maintaining computational efficiency.
2Measurement precision
If complete block matching is performed, then deduplication accuracy is improved, but storage efficiency deteriorates due to redundant storage of similar but not identical blocks
Solution Approach 1:
By segmenting blocks into sub-blocks and identifying duplicate sub-blocks independently, the system can store only the unique portions of similar blocks. This allows partial deduplication where identical sub-blocks are deduplicated while unique sub-blocks are preserved, improving storage efficiency without sacrificing accuracy.
Solution Approach 2:
The patent applies partial deduplication by identifying and deduplicating only the duplicate sub-blocks within candidate blocks rather than requiring complete block matches. This partial action on sub-blocks achieves both accuracy in detecting similarities and efficiency in reducing redundant storage.
3Loss of substance
If sub-block granularity deduplication is implemented, then storage efficiency is improved, but processing complexity increases
Solution Approach 1:
The system computes entropy values for sub-blocks in a self-service manner during the normal data ingestion process. The entropy calculation is integrated into the deduplication workflow itself, eliminating the need for separate complex processing steps and reducing overall system complexity while achieving sub-block granularity deduplication.
Data Source
AI summary
Techniques for processing data may include: receiving a candidate data block; computing a distance using a distance function, wherein the distance is an entropy-based distance and denotes a measurement of similarity between the candidate data block and a target data block; and determining, using the distance, whether to perform data deduplication of the candidate data block with respect to the target data block to identify at least one sub-block of the candidate data block that is a duplicate of at least one sub-block of the target data block. If the distance is less than a threshold, it may be expected to have a matching sub-block between the candidate and target data blocks. The distance may be a difference between entropy values for the candidate and target data blocks. The first entropy value may be used to determine whether to compress or perform partial deduplication for the candidate data block.