Adaptive Hash Key Sizing for Compressibility-Aware Data Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face inefficiencies in data deduplication and compression due to fixed hash key sizes and functions, which do not adapt to the compressibility of data chunks, leading to suboptimal storage utilization and cache management.
Innovation Solution
A method that determines the compressibility of data chunks using entropy values or compression ratios to select varying hash key sizes and functions, optimizing hash table storage and deduplication processing by using smaller hash keys for compressible data and larger keys for less compressible data, thereby improving storage efficiency and cache utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If fixed hash key sizes are used for all data chunks, then the data deduplication system is simple to implement, but storage efficiency deteriorates because larger hash keys are used for highly compressible data than necessary
Solution Approach 1:
The patent applies dynamics by making hash key sizes variable rather than fixed. The system dynamically adjusts hash key sizes based on the compressibility characteristics of each data chunk, using entropy calculation to determine the appropriate size. This resolves the contradiction by allowing the system to use smaller hash keys for highly compressible data (improving storage efficiency) while maintaining a unified variable-size approach that doesn't significantly increase implementation complexity.
Solution Approach 2:
The patent changes the parameter of hash key size from a fixed value to a variable value based on data characteristics. By calculating entropy for each data chunk and using that metric to select an appropriate hash key size, the system optimizes storage efficiency without requiring completely separate processing paths, thus balancing simplicity and performance.
2Quantity of substance
If variable hash key sizes are used based on data compressibility, then storage efficiency is improved, but device complexity increases due to multiple hash functions and size selections
Solution Approach 1:
The patent manages complexity by systematically varying parameters (hash key size and function selection) based on a single metric - entropy. Rather than implementing multiple independent processing paths, the system uses entropy calculation as a unified decision point that determines both hash key size and function selection, making the variable approach more manageable.
Solution Approach 2:
The patent applies preliminary action by calculating entropy for each data chunk before selecting hash parameters. This pre-calculation step provides the necessary information to choose appropriate hash key sizes and functions in advance, organizing the complexity into a structured sequence rather than ad-hoc decisions during deduplication processing.
3Device complexity
If uniform hash table storage is used, then cache management is simple, but cache utilization deteriorates because hash tables allocate fixed space regardless of actual data chunk characteristics
Solution Approach 1:
The patent applies dynamics to cache management by making hash table storage allocation variable based on data chunk compressibility. Highly compressible data chunks use smaller hash keys and occupy less cache space, while less compressible data uses larger hash keys. This dynamic allocation improves cache utilization without requiring completely complex management structures.
Solution Approach 2:
The patent applies local quality by allowing different parts of the cache (hash table entries) to have different sizes based on local data characteristics. Each hash table entry is sized according to the entropy of its associated data chunk, creating locally optimized storage that collectively improves overall cache utilization.
Data Source
AI summary
Techniques for data processing may include: receiving a data chunk; determining a metric value denoting a degree of compressibility of the data chunk; selecting, in accordance with the metric value denoting the compressibility of the data chunk, a first size of a plurality of sizes, wherein each of the plurality of sizes denotes a different size of an amount of storage used for storing a value of said each size; and performing the data deduplication processing for the data chunk, wherein the data deduplication processing includes using a first hash value for the data chunk to determine whether the data chunk is a duplicate of another data chunk of a hash table, wherein the first hash value is stored in a storage location of the first size.