Small-Block Deduplication Using Approximate Hash Signatures
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current deduplication systems face challenges in achieving scalability and efficiency when dealing with small data chunks, as they struggle to balance uniformity and sensitivity in hash functions, leading to issues with storage capacity and collisions.
Innovation Solution
An approximate hash function is developed that combines character and frequency representations of data chunks, using a signature generated from character spectra and pairs, which is less sensitive to small changes while maintaining uniform distribution, allowing for effective deduplication of both identical and similar blocks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional hash functions are used for deduplication, then collision detection is improved, but sensitivity to small changes causes excessive collisions and reduced uniformity
Solution Approach 1:
The data chunk is divided into multiple substrings of equal length, and each substring is processed independently to generate a component hash value. This segmentation reduces the sensitivity to small changes in the original data while maintaining collision detection capability through the combination of component hashes.
Solution Approach 2:
The patent changes the parameter of hash function sensitivity by using substring-based processing instead of processing the entire chunk at once. This parameter change achieves a balance between uniformity and sensitivity by making the hash function less sensitive to small changes while maintaining collision detection through the aggregation of multiple substring hashes.
2Quantity of substance
If fine-grained data blocks are used for deduplication, then storage efficiency is improved, but system complexity and processing overhead increase
Solution Approach 1:
The patent segments fine-grained data blocks into equal-length substrings for hash computation. This segmentation approach enables efficient processing of small data chunks by breaking them into manageable pieces, reducing the computational complexity while maintaining the storage efficiency benefits of fine-grained deduplication.
Solution Approach 2:
The patent uses a simplified hash computation approach that copies and processes only the necessary substring portions of data rather than entire data blocks. This reduces the processing overhead and system complexity while maintaining the ability to detect duplicates at fine-grained levels.
3Manufacturing precision
If small data chunks are processed, then deduplication resolution is improved, but processing speed and performance decrease
Solution Approach 1:
The patent segments small data chunks into equal-length substrings that can be processed in parallel or in a streamlined sequence. This segmentation improves processing efficiency by enabling batch processing of uniform-sized pieces, maintaining high deduplication resolution while improving overall processing throughput.
Solution Approach 2:
The patent changes the processing parameter from handling variable-sized small chunks to processing fixed-size substrings. This parameter change optimizes processing speed by enabling more efficient memory access patterns and parallel processing, while maintaining fine-grained deduplication resolution through the substring approach.
Data Source
AI summary
Exemplary method, system, and computer program product embodiments for scalable data deduplication working with small data chunk in a computing environment are provided. In one embodiment, by way of example only, for each of the small data chunk, a signature is generated based on a combination of a representation of characters that appear in the small data chunk with a representation of frequencies of the small data chunk. A signature is generated based on a combination of a representation of characters that appear. The signature is used to help in selecting the data to be deduplicated. Additional system and computer program product embodiments are disclosed and provide related advantages.