Recursive Data Deduplication via Multi-Granularity Chunk Scanning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In deduplication systems, deduplicating data at a fixed smaller length leads to increased metadata and fragmented data, resulting in degraded performance and reduced storage capacity due to unnecessary deduplication of data chunks.
Innovation Solution
The system scans data extents at various chunk lengths and shift positions to identify larger duplicate data chunks, using an Extent Family with offset references to represent smaller chunks, thereby reducing metadata and fragmentation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If data is deduped at a fixed smaller length, then duplicate data can be identified at the smaller chunk level, but metadata and fragmented data increase, resulting in degraded performance
Solution Approach 1:
The patent divides the deduplication process into multiple levels based on chunk sizes. It segments data into different chunk size categories (e.g., small chunks, medium chunks, large chunks) and applies different deduplication strategies to each segment. This allows the system to identify duplicates at appropriate granularities without uniformly applying fine-grained deduplication to all data, thereby reducing metadata overhead while maintaining duplicate detection accuracy.
Solution Approach 2:
The patent applies different deduplication qualities to different parts of the data based on local characteristics. Instead of using a uniform deduplication approach, it analyzes data patterns and applies appropriate chunk sizes locally - using smaller chunks where precision is needed and larger chunks where it reduces metadata without losing duplicate detection effectiveness. This local adaptation optimizes the balance between duplicate identification precision and performance.
2Measurement precision
If data is deduped at a fixed smaller length, then duplicate data can be identified at the smaller chunk level, but metadata increases, resulting in reduced storage capacity
Solution Approach 1:
The patent segments the metadata structure into hierarchical levels corresponding to different chunk sizes. By organizing metadata to reflect the multi-level chunk segmentation, the system avoids creating redundant metadata entries for every possible small chunk combination. Instead, it creates metadata structures that efficiently represent duplicates at appropriate granularities, reducing overall metadata quantity while preserving duplicate identification capability.
Solution Approach 2:
The patent introduces a dimensional approach to metadata organization by adding chunk size as an explicit dimension in the metadata structure. This allows the system to index and manage duplicates across multiple size dimensions simultaneously, reducing the need for excessive metadata entries by organizing information in a multi-dimensional space rather than a single flat structure.
3Measurement precision
If data is deduped at a fixed smaller length, then duplicate data can be identified at the smaller chunk level, but fragmented data increases, resulting in degraded performance
Solution Approach 1:
The patent implements segmentation of the deduplication process by chunk size, which naturally reduces fragmentation by grouping related data segments together. Instead of creating numerous small fragmented deduplication units, it segments data into logical chunks of appropriate sizes and processes them as cohesive units, thereby reducing the fragmentation of both data and metadata structures.
Solution Approach 2:
The patent inverts the conventional approach by not starting with fixed small chunks and merging them, but rather by identifying larger natural data boundaries and then determining appropriate chunk sizes from those boundaries. This inversion prevents artificial fragmentation by allowing data structure to dictate chunk boundaries rather than imposing uniform small boundaries on heterogeneous data.
Data Source
AI summary
A virtual computing system deduplicates data. The system includes one or more processors, and one or more computer-readable storage media communicably coupled to the one or more processors. The one or more computer-readable storage media has instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to: scan extents for duplicate data chunks; calculate hash values for the extents at different chunk lengths; emit the hash values at the different chunk lengths; identify matching ones of the hash values; and select one or more candidates from among the matching ones of the hash values for dedup.