Hash Table Structure for Digest Matching in Data Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current deduplication systems face challenges in scaling to support large repositories of data, such as Petabytes, due to high resource consumption and storage requirements for fingerprint indexes, leading to performance bottlenecks and increased costs, especially when performing byte-wise data comparisons.
Innovation Solution
A two-step deduplication approach using a similarity search to focus on matching digests, where input data is segmented into chunks, and similarity elements and digest block boundaries are calculated using rolling hash values, reducing the need for high I/O rates and shifting resource consumption from disks to CPUs, allowing for efficient matching of similar data patterns without extensive byte-wise comparisons.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If byte-wise data comparisons are performed for deduplication, then matching accuracy is improved, but resource consumption and processing time increase significantly
Solution Approach 1:
The patent segments data into fixed-size chunks and computes rolling hash values for each chunk. This segmentation allows the system to process data in manageable units and use hash-based pre-filtering to identify candidate matches before performing byte-wise comparisons, thereby reducing overall processing time while maintaining matching accuracy.
Solution Approach 2:
The patent performs preliminary rolling hash computations and similarity element calculations before actual deduplication matching. By pre-computing hash values and organizing data with similarity indexes, the system reduces the scope of subsequent byte-wise comparisons, improving processing efficiency without sacrificing matching precision.
2Quantity of substance
If large repositories of data are supported, then storage capacity is improved, but I/O demands and performance bottlenecks increase
Solution Approach 1:
The patent replaces mechanical I/O-intensive byte-wise comparisons with CPU-based rolling hash computations and similarity element calculations. This substitution shifts the workload from disk I/O to CPU processing, enabling the system to handle large repositories without proportionally increasing I/O demands.
Solution Approach 2:
The patent introduces rolling hash values and similarity elements as intermediary data structures between the raw data and the deduplication matching process. These intermediaries enable efficient candidate identification and reduce the need for extensive I/O operations during actual matching.
3Loss of energy
If resource consumption is shifted from disks to CPUs, then I/O demands are reduced, but processing architecture complexity increases
Solution Approach 1:
The patent implements a universal processing framework where rolling hash computations serve multiple purposes: generating similarity elements for indexing, creating hash values for candidate identification, and enabling efficient deduplication matching. This multi-functionality reduces the need for separate processing systems while achieving I/O reduction.
Data Source
AI summary
Repository data intervals are determined as similar to an input data interval. Repository digests corresponding to the similar repository data interval are loaded into a sequential representation and into a search structure. Matches of input digests and the repository digests are found using the search structure. Each one of the found matches of the input digests and repository digests are extended using the sequential representation. Data matches are determined between the input data and the repository data using extended matches of digests. A compact index pointing to a position in the sequential representation of digests is incorporated into entries of the search structure.