Similarity Hash Computation for Partial Data Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data deduplication methods fail to detect redundant data when blocks differ by only a single bit or byte, missing opportunities for deduplication of similar but not identical data blocks.
Innovation Solution
The method involves determining a similarity hash for data blocks by partitioning them into features, using precomputed hash values to generate feature hashes, and then calculating a similarity hash to identify similar blocks, allowing for partial deduplication and compression of similar but not identical data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional exact-match deduplication methods are used, then storage efficiency is improved for identical blocks, but similar blocks that differ by even a single bit or byte are not detected, missing deduplication opportunities
Solution Approach 1:
The data block is divided into multiple features (e.g., bytes or words), and each feature is processed independently to generate a feature hash. This segmentation allows the system to identify partial similarities between blocks by comparing individual feature hashes, enabling detection of similar blocks that differ in only some features rather than requiring exact matches across the entire block.
Solution Approach 2:
The system changes the parameter of hash comparison from requiring exact equality to allowing partial matches based on a threshold. By computing a similarity score based on the number of matching feature hashes and comparing it against a threshold, the system can identify similar blocks even when they differ by some bits or bytes, thereby improving both detection accuracy and storage utilization.
2Productivity
If precomputed hash tables are used for all possible feature values, then feature hash lookup efficiency is improved, but memory consumption increases
Solution Approach 1:
Instead of computing all possible feature hashes for every possible input value, the system precomputes and stores only the feature hashes that are actually needed based on the specific data characteristics. This partial precomputation approach maintains fast lookup performance for relevant cases while avoiding the excessive memory consumption of storing all possible hash values.
Solution Approach 2:
The precomputed hash table is designed to serve multiple features simultaneously, where each entry can be used for comparing different features across multiple data blocks. This multi-functional design allows the same precomputed values to be reused in various comparison operations, reducing overall memory requirements while maintaining computational efficiency.
Data Source
AI summary
Techniques for data processing may include: receiving a data block P having a binary representation; determining features for the data block P; determining, using at least one table of precomputed hash values, feature hashes for the features, wherein each of the feature hashes corresponds to a different feature, wherein each of the feature hashes is one of the precomputed hash values of the at least one table; and determining, in accordance with the feature hashes, a similarity hash for the data block P. Each feature may be a byte of P. The at least one table may be a single 3 dimensional or multiple 2 dimensional tables. Each row of a table of precomputed hash values may correspond to a single precomputed hash value. The row may include byte entries where each byte entry includes a single bit value of a precomputed hash.