Deduped Clump Layout for Storage Index Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication systems face challenges in managing large fingerprint indices in distributed storage systems, particularly in virtualized environments, where random access and snapshot retention lead to inefficient storage and performance issues due to the need for extensive RAM and poor space efficiency.
Innovation Solution
The Deduped Clump Layout (DCL) approach groups data blocks into larger clumps, which are then deduplicated, reducing the number of index entries and allowing the fingerprint index to fit in RAM, while maintaining efficient storage and performance by proactive clump repacking and garbage collection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If block-level deduplication is used, then storage efficiency is improved, but the index size becomes too large to fit in RAM
Solution Approach 1:
The patent segments the storage space into fixed-size extents (e.g., 64 KB) and creates index entries at the extent level rather than block level. This segmentation reduces the number of index entries from potentially millions of blocks to a manageable number of extents, allowing the index to fit in RAM while maintaining deduplication capability.
Solution Approach 2:
The patent introduces a hierarchical indexing structure with multiple levels. The first level indexes extents, and subsequent levels provide finer granularity when needed. This dimensional approach allows the system to balance between index size and deduplication efficiency by navigating through different levels of abstraction.
2Quantity of substance
If larger extents are used for deduplication, then the index size is reduced, but space efficiency deteriorates when only small portions of extents differ
Solution Approach 1:
The patent further segments extents into smaller sub-extents or blocks within each extent. When deduplication is performed, the system can identify and store only the differing portions (sub-extents) rather than the entire extent, improving space efficiency while keeping the extent-level index manageable.
Solution Approach 2:
The patent applies different levels of granularity to different parts of the storage hierarchy. Extents are used for indexing to keep the index small, while sub-extents or blocks within extents are used for actual deduplication to maximize space efficiency. This local quality approach optimizes both index size and deduplication effectiveness.
3Device complexity
If fixed-size extents are used, then indexing is simplified, but performance deteriorates due to read-modify-write operations on entire extents
Solution Approach 1:
The patent performs preliminary actions by pre-processing extents to identify and extract only the modified sub-extents or blocks before writing. This allows the system to maintain fixed-size extents for simple indexing while avoiding the performance penalty of reading and writing entire extents by preparing the data in advance.
Solution Approach 2:
The patent extracts only the modified portions (sub-extents or blocks) from the larger extents for storage and indexing purposes. This extraction approach allows the system to use fixed-size extents for indexing simplicity while storing and managing only the necessary changed data, improving write performance.
Data Source
AI summary
Data in a storage system is deduplicated after receiving from at least one writing entity requests for a plurality of write operations for a corresponding plurality of data blocks in a storage object. The received blocks are buffered and sorted in order and a sequence of clumps is created from the buffered blocks, where each clump comprises a grouping of at least one of the sorted, buffered blocks. A boundary is determined between at least one pair of clumps based at least in part on the content of at least one of the buffered blocks, and it is then determined whether at least one of the clumps is a duplicate of a previously stored clump.