In-line Deduplication Hash Indexing for Cloud Storage Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current in-line deduplication techniques in cloud storage systems face inefficiencies due to high computational costs and memory intensity, leading to significant time delays and performance penalties, especially for large objects, which can result in bottlenecks and denial-of-service conditions.
Innovation Solution
The method involves calculating hash values in iterative steps on fixed-size data chunks of objects, matching these values with existing objects in a hash index table, and updating tables to determine unique objects early in the process, thereby reducing the need for full-object hashing and minimizing resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If full-object hashing is performed for in-line deduplication, then deduplication accuracy is improved, but computational cost and processing time increase significantly
Solution Approach 1:
The patent divides the object into multiple fixed-size data chunks and calculates hash values for each chunk separately. This segmentation allows the system to process only portions of the object rather than the entire object, significantly reducing hash calculation time while maintaining deduplication accuracy through iterative matching.
Solution Approach 2:
The patent performs preliminary hash calculations on data samples from each data chunk before performing full object hashing. By pre-calculating hashes of individual chunks and storing them in hash index tables, the system can quickly determine uniqueness without always requiring full-object hashing, thus reducing overall processing time.
2Productivity
If iterative hash calculation on data chunks is performed, then processing speed is improved, but system complexity increases
Solution Approach 1:
The patent creates multiple hash index tables corresponding to different data chunks (first data chunk hash index table, second data chunk hash index table, etc.). This segmentation of the indexing structure allows parallel processing of different chunks while maintaining manageable complexity through modular organization.
Solution Approach 2:
The patent performs hash calculations and table lookups in iterative cycles, processing one data chunk at a time in sequence. This periodic action breaks down the complex task of full-object processing into simpler, repetitive steps that are easier to manage and execute efficiently.
3Loss of energy
If early exit on mismatch is implemented, then resource utilization is reduced, but risk of false negatives increases
Solution Approach 1:
The patent performs preliminary hash calculations on data samples from each data chunk and stores them in hash index tables before final deduplication determination. This preliminary action creates a reference framework that enables early exit on mismatch while maintaining reliability, as the pre-calculated hashes serve as reliable indicators of uniqueness.
Solution Approach 2:
The patent uses feedback from hash value comparisons to determine whether to continue or exit the deduplication process. When a mismatch is detected in the hash index tables, the system receives feedback indicating uniqueness and can confidently exit early. The feedback mechanism ensures that early exit decisions are based on reliable hash comparison results rather than random termination.
Data Source
AI summary
Exemplary embodiments for increased in-line deduplication efficiency in a computing environment are provided. In one embodiment, by way of example only, hash values are calculated in nth iterations on data samples from fixed size data chunks extracted from an object requested for in-line deduplication. For each of the nth iterations, the calculated hash values for the data samples from the fixed size data chunks are matched in an nth hash index table with a corresponding hash value of existing objects in storage. The nth hash index table is exited upon detecting a mismatch during the matching. The mismatch is determined to be a unique object and is stored. A hash value for the object is calculated. A master hash index table is updated with the calculated hash value for the object and the calculated hash values for the unique object.