Sketch Computation for Inline Data Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional deduplication techniques are inefficient in handling large-scale data reduction, particularly for enterprise applications, due to the computational burden and memory requirements associated with generating and comparing large index tables, making it impractical for petabyte-scale data management.
Innovation Solution
The method involves generating a 'sketch' of data segments using hashing algorithms, which represents the data with a minimal set of features for inline deduplication, allowing for efficient comparison and storage without the need for a full index or chunk-by-chunk comparison, thereby reducing computational resources and storage needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional deduplication techniques use full index generation and chunk-by-chunk comparison, then data reduction accuracy is improved, but computational burden and memory requirements increase significantly
Solution Approach 1:
The patent segments the data processing task by dividing incoming data streams into fixed-size segments, then further divides each segment into variable-size chunks. This segmentation allows the system to process data in manageable units and apply different processing strategies at different levels, reducing the overall computational burden while maintaining deduplication accuracy.
Solution Approach 2:
The patent introduces sketches as an intermediary data structure between the raw data chunks and the full index. Sketches are compact probabilistic summaries that enable approximate deduplication detection without requiring complete chunk-by-chunk comparison. This intermediary layer significantly reduces memory requirements and computational complexity while maintaining acceptable data reduction accuracy.
2Measurement precision
If conventional deduplication techniques generate large index tables for petabyte-scale data, then deduplication coverage is improved, but memory space consumption increases
Solution Approach 1:
The patent extracts only the essential features of data segments into compact sketches rather than storing complete chunk information in the index. By taking out only the necessary identifying characteristics and storing them in a compressed sketch format, the system achieves broad deduplication coverage with minimal memory space consumption.
Solution Approach 2:
The patent uses inexpensive sketch data structures that can be quickly created and discarded, replacing the need for persistent, memory-intensive full indexes. These sketch objects serve their purpose of enabling deduplication detection and then can be freed, allowing the system to handle petabyte-scale data with limited memory resources.
3Measurement precision
If conventional deduplication techniques perform comprehensive chunk comparisons, then deduplication completeness is improved, but processing throughput decreases
Solution Approach 1:
The patent performs preliminary processing by generating sketches and computing segment hashes before actual deduplication comparison. This preliminary action filters out obviously duplicate segments early in the process, avoiding the need for expensive chunk-by-chunk comparisons in all cases and thereby maintaining high processing throughput while preserving deduplication completeness.
Solution Approach 2:
The patent applies partial comparison strategies by using sketches to perform approximate deduplication detection first, then applying full chunk-by-chunk comparison only when necessary. This partial action approach maintains high processing throughput by avoiding unnecessary comprehensive comparisons while ensuring deduplication completeness through selective full verification.
Data Source
AI summary
A method for sketch computation is provided. The method may comprise receiving an input data stream from one or more client applications. The method may also comprise generating at least one segment from the input data stream. At least one segment may comprise a plurality of chunks. The method may further comprise computing a sketch of the at least one segment. The sketch may comprise a set of features that are representative of or unique to the at least one segment, such that the set of features corresponds to the at least one segment. The sketch may be useable for inline deduplication of at least one other input data stream received from the one or more client applications without (i) generation of a full index of the plurality of chunks or (ii) comparison of the at least one other input data stream to the full index.