Chunk-Level Deduplication Using Representative Digests
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional deduplication schemes are inefficient when handling data received in increments larger than a standard block size, leading to unnecessary metadata storage, as they store digests for each block even if the entire chunk of data is a duplicate.
Innovation Solution
Perform deduplication at a coarser granularity by grouping multiple blocks into chunks and using a representative subset to generate a hash digest for the entire chunk, allowing for deduplication at a chunk level, and utilizing multiple digest databases for different chunk sizes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional block-level deduplication is used on data received in large increments (e.g., 64-KB chunks), then each block within the chunk is individually hashed and stored in the metadata database, but this results in excessive metadata storage when the entire chunk is a duplicate
Solution Approach 1:
The patent segments the deduplication process into two levels: chunk-level segmentation for initial duplicate detection using representative digests, and block-level segmentation for detailed verification when needed. This hierarchical segmentation allows the system to efficiently handle large data increments by first determining at the chunk level whether duplication exists, thereby avoiding unnecessary block-level processing and metadata storage for entirely duplicate chunks.
Solution Approach 2:
The patent extracts a representative subset of blocks from each chunk to generate a representative digest for chunk-level deduplication. By taking out only the necessary representative blocks rather than processing all blocks, the system reduces metadata storage requirements while maintaining deduplication effectiveness for large data increments.
2Productivity
If block-level deduplication is performed on large data increments, then individual block digests are computed and stored, but this increases processing overhead and reduces deduplication efficiency
Solution Approach 1:
The patent performs preliminary chunk-level deduplication by computing representative digests before proceeding to block-level processing. This preliminary action allows the system to quickly identify and eliminate entirely duplicate chunks, thereby reducing the overall processing time and improving deduplication efficiency for large data increments by avoiding unnecessary block-level operations.
Solution Approach 2:
The patent applies partial action by computing digests for only a representative subset of blocks rather than all blocks in a chunk. This partial processing approach maintains adequate deduplication efficiency while significantly reducing the processing time and computational overhead associated with block-level deduplication of large data increments.
3Adaptability or versatility
If standard block-size deduplication is used for applications writing in larger increments, then the system stores metadata for each block, but this creates a mismatch between application write patterns and deduplication granularity
Solution Approach 1:
The patent introduces dynamic granularity by allowing the deduplication system to adapt between chunk-level and block-level processing based on the specific data characteristics and application write patterns. This dynamic approach improves adaptability to different application requirements while managing metadata complexity through automated selection of the appropriate deduplication level.
Solution Approach 2:
The patent creates a universal deduplication framework that can handle both traditional block-level deduplication and the new chunk-level deduplication approach. This multi-functional system adapts to different application write patterns (whether small block writes or large increment writes) while maintaining a unified metadata management structure, thereby improving versatility without proportionally increasing complexity.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach reduces unnecessary metadata storage by efficiently identifying and eliminating duplicate chunks of data, improving deduplication efficiency even when data is received in larger increments.
Implementation Method 1
hashing a subset of the set of data, yielding a representative digest of the set of data
Data Source
AI summary
A method of performing deduplication includes (1) receiving a write command that specifies a set of data, the set of data including multiple blocks of data, (2) hashing a subset of the set of data, yielding a representative digest of the set of data, and (3) performing deduplication on the set of data based at least in part on matching the representative digest to a digest already stored in a database which relates digests to locations of data from which the digests were produced. An apparatus, system, and computer program product for performing a similar method are also provided.