Similarity Deduplication Using Partial Locality-Sensitive Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face challenges in efficiently determining similarity between data units for dictionary-based compression, requiring extensive computational resources and lacking accuracy in exact matching schemes without clustering.
Innovation Solution
The method involves applying a locality-sensitive hash function to only a portion of a data unit to produce a hash value, determining maximum or minimum hash values from chunks, and forming a representative portion by concatenating contiguous pieces, which is then used for dictionary-based compression or self-compression based on hash index matching.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a hash function is applied to the entire data unit to determine similarity, then the accuracy of similarity detection is improved, but the computational resources and time required increase significantly
Solution Approach 1:
The data unit is divided into multiple chunks, and hash functions are applied to each chunk separately rather than to the entire data unit. This segmentation allows the system to process smaller portions of data in parallel, reducing the overall computational time while maintaining similarity detection accuracy through aggregation of chunk hash values.
Solution Approach 2:
The patent applies hash functions to only a portion (selected chunks) of the data unit rather than the complete data unit. By selecting representative chunks that capture the essential characteristics of the data, the system achieves sufficient similarity detection accuracy with reduced computational effort compared to processing the entire data unit.
2Productivity
If dictionary-based compression is used with exact matching schemes, then the data reduction efficiency is improved, but the computational resources required for determining exact matches increase
Solution Approach 1:
Hash values serve as an intermediary mechanism between data units and dictionary-based compression. By computing hash values for data chunks and comparing these compact representations, the system can efficiently identify similar data units without performing expensive exact matching operations, thereby reducing computational resources while maintaining data reduction efficiency.
Solution Approach 2:
The patent replaces the mechanical process of exact matching (byte-by-byte comparison) with a hash-based comparison mechanism. This substitution uses mathematical hash functions to generate compact representations of data chunks, which can be compared much more efficiently than the original data, reducing the computational energy required for similarity determination.
3Reliability
If clustering is applied to improve similarity detection accuracy, then the true positive rate is improved, but the device complexity and computational overhead increase
Solution Approach 1:
The system uses self-organizing properties of hash value distributions to implicitly perform clustering-like functionality. By selecting chunks that produce extreme hash values (minimum or maximum), the system automatically identifies representative portions of data that capture similarity patterns without requiring explicit clustering algorithms, thereby maintaining high true positive rates while reducing system complexity.
Data Source
AI summary
Dictionary-based compression is performed to compress data units using a similar data unit as the base unit (i.e., dictionary) for each candidate data unit. Similarity may be determined between data units by applying a locality-sensitive hashing scheme to each candidate data unit to produce a hash value, and by determining whether there is a matching value in a hash index of hash values for existing data units on the system. If there is a matching hash value, the candidate data unit may be compressed using the data unit corresponding to the matching hash value as the dictionary. Only a representative portion of the data unit may be hashed to produce the hash value, the portion comprised of chunks of the data unit, where each chunk is a continuous, uninterrupted section of data. The chunks themselves may not be (in some embodiments likely are not) contiguous to one another.