Data Segment Sketching for Memory-Efficient Inline Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data deduplication techniques struggle to handle large-scale data reduction efficiently, particularly for enterprise applications, due to the computational burden and memory requirements of generating and comparing fingerprints for data chunks, making them impractical for petabyte-scale data management.
Innovation Solution
The use of sketches to characterize large data segments, comparing these sketches for similarity, and performing differencing operations only when necessary, reduces memory and computational demands by generating pointers for groups of matching chunks instead of individual chunks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If conventional fingerprint-based deduplication techniques are used to identify and eliminate duplicate data chunks, then data reduction capability is improved, but memory consumption and computational burden increase significantly
Solution Approach 1:
The patent divides data into fixed-size chunks and uses probabilistic counting sketches to represent each chunk's fingerprint information. Instead of storing complete fingerprints for all chunks, the system segments the deduplication process into chunk-level processing with compact sketch representations, reducing memory consumption while maintaining deduplication capability.
Solution Approach 2:
The patent transforms the representation of data chunk fingerprints from exact hash values to probabilistic counting sketches. This parameter change allows the system to approximate fingerprint matching with significantly reduced memory requirements, trading exact precision for scalable memory efficiency in handling large-scale data deduplication.
2Measurement precision
If complete fingerprint comparison is performed for every data chunk to ensure accurate deduplication, then deduplication accuracy is improved, but processing time increases
Solution Approach 1:
The patent performs partial fingerprint comparison using counting sketches instead of complete fingerprint verification for all chunks. The system uses probabilistic matching to identify potential duplicates quickly, then performs more rigorous verification only when necessary, reducing overall processing time while maintaining acceptable deduplication accuracy.
Solution Approach 2:
The patent uses lightweight, disposable counting sketch structures that can be rapidly created and compared. These temporary sketch representations serve as efficient proxies for complete fingerprints, enabling fast preliminary comparisons without the computational overhead of full fingerprint verification for every data chunk.
3Loss of substance
If an index table is generated to store all data chunk fingerprints for deduplication, then deduplication functionality is improved, but system complexity and memory overhead increase
Solution Approach 1:
The patent extracts only the essential fingerprint information from complete data chunks and stores it in compact counting sketch structures. Instead of maintaining comprehensive index tables with full fingerprint details, the system extracts and stores only the necessary probabilistic representation, simplifying the overall system architecture while preserving deduplication functionality.
Solution Approach 2:
The patent creates simplified copies of fingerprint information in the form of counting sketches. These sketch copies serve as lightweight proxies that capture the essential characteristics needed for deduplication without replicating the full complexity of original fingerprint data structures, reducing system complexity and memory overhead.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method for sketch computation is provided. The method may comprise receiving an input data stream from one or more client applications. The method may also comprise generating at least one segment from the input data stream. At least one segment may comprise a plurality of chunks. The method may further comprise computing a sketch of the at least one segment. The sketch may comprise a set of features that are representative of or unique to the at least one segment, such that the set of features corresponds to the at least one segment. The sketch may be useable for inline deduplication of at least one other input data stream received from the one or more client applications without (i) generation of a full index of the plurality of chunks or (ii) comparison of the at least one other input data stream to the full index.