Block-Level Deduplication Using Express Directory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Block-level deduplication is resource-intensive due to the need for extensive processing to compare data blocks for duplicates, leading to high storage and processing requirements, especially with smaller block sizes, which consume valuable resources in storage systems.
Innovation Solution
Implementing an enhanced deduplication system that uses an express directory for frequently duplicated block signatures, computed and stored in high-speed memory, allowing for quick matching and bit-by-bit comparisons only when necessary, while maintaining a primary directory for all block signatures with reference counts, and adjusting the reference count requirement based on directory size to optimize resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If block-level deduplication is performed with smaller block sizes, then deduplication efficiency is improved, but storage requirements for directories and processing resources increase significantly
Solution Approach 1:
The patent divides the deduplication system into two distinct components: a primary directory that stores all block signatures and reference counts, and a secondary directory that stores only frequently duplicated block signatures. This segmentation allows the system to handle smaller block sizes efficiently while minimizing the storage overhead of the secondary directory by only containing high-frequency duplicates.
Solution Approach 2:
The patent applies local quality by creating a specialized secondary directory with different storage characteristics optimized for specific purposes (storing frequently accessed duplicate signatures). This directory has different access patterns and storage requirements compared to the primary directory, allowing optimized resource allocation where high-frequency duplicates are kept in faster, more accessible storage while less frequent duplicates remain in the primary directory.
2Productivity
If block-level deduplication is performed with smaller block sizes, then deduplication efficiency is improved, but processing resources consumed increase
Solution Approach 1:
The patent performs preliminary action by pre-computing and storing block signatures in the primary directory before deduplication operations are needed. When deduplication is performed, the system can quickly reference pre-computed signatures rather than computing them on-demand, significantly reducing processing resources during actual deduplication operations while maintaining support for smaller block sizes.
Solution Approach 2:
The patent applies partial action by implementing a secondary directory that stores only a subset of block signatures - specifically those that are frequently duplicated. This partial storage approach reduces the overall processing burden compared to storing all signatures, while still capturing the majority of deduplication opportunities through the high-frequency duplicate tracking.
3Productivity
If background processes perform signature comparing and bit by bit comparing, then deduplication is achieved, but valuable storage system resources are consumed
Solution Approach 1:
The patent performs preliminary signature comparison by pre-storing all block signatures in the primary directory with their reference counts. This allows the system to quickly determine potential duplicates through signature matching before performing resource-intensive bit-by-bit comparisons, reducing the overall processing burden on background processes and freeing up system resources for other operations.
Solution Approach 2:
The patent introduces an intermediary mechanism - the secondary directory - that acts as a mediator between the primary directory and the bit-by-bit comparison process. The secondary directory pre-identifies frequently duplicated blocks, allowing the system to skip unnecessary bit-by-bit comparisons for blocks that are unlikely to be duplicates, thereby reducing resource consumption while maintaining deduplication accuracy.
Data Source
AI summary
An apparatus, system, and method are disclosed for enhanced block-level deduplication. A computer program product stores one or more express block signatures in an express directory that meet a reference count requirement. The computer program product also stores one or more primary block signatures and one or more reference counts for the primary block signatures in a primary directory. Each primary block signature has a corresponding reference count. The computer program product determines whether a block signature for a data block matches one of the one or more express block signatures stored in the express directory.