Data Deduplication via Sketch Computation for Petabyte Scale Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional deduplication techniques are inefficient in handling large-scale data reduction, particularly for enterprise applications, due to the computational burden and memory requirements associated with generating and comparing large index tables, making them impractical for petabyte-scale data management.
Innovation Solution
The method involves generating a 'sketch' of data segments using hashing algorithms, which represents the data with a minimal set of features that can be used for inline deduplication without requiring a full index or chunk-by-chunk comparison, allowing for efficient data reduction and storage on a petabyte scale.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional deduplication techniques use full index tables and 1:1 chunk comparisons, then data reduction accuracy is improved, but computational burden and memory space consumption increase significantly
Solution Approach 1:
The patent segments the data into fixed-size chunks and organizes them into an index table structure, dividing the large-scale data processing task into manageable chunk-level operations. This segmentation enables efficient indexing and comparison while reducing the computational complexity of handling entire data sets at once.
Solution Approach 2:
The patent introduces an index table as an intermediary data structure that stores chunk identifiers and metadata. This index table acts as a mediator between the raw data and the deduplication comparison process, enabling efficient lookup and comparison operations without requiring direct processing of the entire data set, thereby reducing computational burden.
2Reliability
If conventional deduplication techniques generate full index tables, then deduplication completeness is improved, but memory space consumption increases significantly
Solution Approach 1:
The patent extracts only the essential identifying features of each chunk (such as chunk identifiers and key metadata) and stores them in the index table, rather than storing complete copies of the data. This extraction approach maintains deduplication completeness by preserving sufficient information for accurate comparison while dramatically reducing memory space consumption.
Solution Approach 2:
The patent creates a simplified copy of the data structure in the form of an index table that contains references and metadata rather than full data copies. This copying strategy enables the system to maintain deduplication functionality while using minimal memory resources, as the index table serves as a lightweight representation of the actual data.
3Measurement precision
If conventional deduplication techniques perform 1:1 chunk comparisons, then deduplication accuracy is improved, but processing speed decreases
Solution Approach 1:
The patent performs preliminary organization of data into chunks and pre-computation of chunk identifiers and metadata before the actual deduplication comparison. This preliminary action prepares the data in an optimized format that enables faster comparison operations, maintaining accuracy while improving processing speed by avoiding repeated data preparation during comparison.
Solution Approach 2:
The patent replaces the mechanical process of direct 1:1 chunk comparison with a more efficient lookup-based approach using the index table. Instead of comparing entire chunks sequentially, the system uses the pre-built index structure to quickly identify potential duplicates through identifier matching, significantly improving processing speed while maintaining deduplication accuracy.
Data Source
AI summary
A method for data reduction may comprise computing (i) a first sketch of a first segment and (ii) a second sketch of a second segment. The first sketch and the second sketch may each comprise a set of features that are representative of or unique to the corresponding first and second segments. The method also comprise processing the first sketch and the second sketch to generate a similarity metric indicative of whether the second segment is similar to the first segment. The method may further comprise (1) performing a differencing operation on the second segment relative to the first segment when the similarity metric is greater than or equal to a similarity threshold, or (2) storing the first segment and the second segment in a database without performing the differencing operation when the similarity metric is less than the similarity threshold.