Approximate Hashing for Small-Block Deduplication Scalability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current deduplication systems face challenges in achieving scalability and efficiency when dealing with small data chunks, as they struggle to balance uniformity and sensitivity in hash functions, leading to issues with storage capacity and collisions.
Innovation Solution
The implementation of an approximate hash scheme that generates a signature based on character and frequency distributions, allowing for the detection of both identical and similar data blocks, using a combination of character and frequency spectra to create a 32-bit hash value that is resistant to small changes, thereby improving deduplication efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional hash functions are used for deduplication, then collision resistance is improved, but sensitivity to small changes causes excessive collisions in small data blocks
Solution Approach 1:
The patent segments the hash function into two distinct components: a primary hash function for collision resistance and a secondary sensitivity-reduction function. This segmentation allows each component to optimize for its specific purpose while working together to solve the overall contradiction.
Solution Approach 2:
The patent changes the parameters of the hash function by introducing a new transformation step that modifies how small changes in input affect the output. This parameter change reduces the sensitivity to small variations while preserving the collision resistance properties of the original hash function.
2Quantity of substance
If fine-grained deduplication of small blocks is implemented, then storage efficiency is improved, but system scalability deteriorates
Solution Approach 1:
The patent applies partial action by using a simplified hash comparison approach for small data blocks rather than full-block comparison. This partial approach achieves sufficient deduplication efficiency for small blocks without requiring the complete scalability infrastructure needed for larger blocks.
Solution Approach 2:
The patent applies local quality by implementing different deduplication strategies for different block sizes. Small blocks use the specialized hash function with reduced sensitivity, while larger blocks use traditional approaches, allowing the system to optimize for storage efficiency locally without compromising overall scalability.
3Reliability
If uniform hash distribution is achieved, then collision rate is reduced, but sensitivity to small changes increases causing false negatives
Solution Approach 1:
The patent introduces an intermediary transformation function that sits between the uniform hash distribution and the final comparison. This intermediary reduces the sensitivity to small changes while preserving the uniform distribution properties, acting as a mediator that balances both requirements.
Solution Approach 2:
The patent creates a composite hash function by combining a uniform-distribution hash function with a sensitivity-reduction transformation. This composite approach leverages the strengths of both components: the uniform distribution for collision resistance and the transformation for reduced sensitivity to small changes.
Data Source
AI summary
Exemplary method, system, and computer program product embodiments for scalable data deduplication working with small data chunk in a computing environment are provided. In one embodiment, by way of example only, for each of the small data chunk, a signature is generated based on a combination of a representation of characters that appear in the small data chunk with a representation of frequencies of the small data chunk. A signature is generated based on a combination of a representation of characters that appear. The signature is used to help in selecting the data to be deduplicated. Additional system and computer program product embodiments are disclosed and provide related advantages.