Data Deduplication Using Augmented Cuckoo Filters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data deduplication systems in information handling systems face inefficiencies in storage space conservation and processing capacity, particularly due to the reliance on collision-resistant hashing algorithms and the need for extensive RAM resources, which limits their ability to recognize unique data blocks quickly and conserve storage capacity effectively.
Innovation Solution
The implementation of a two-tiered data dictionary system that includes block hash to storage location mappings in the first tier and augmented cuckoo filters in the second tier, allowing for efficient querying and verification of duplicate data blocks using fingerprints derived from block hashes, thereby reducing the need for extensive processing capacity and RAM resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If collision-resistant hashing algorithms are used to ensure data integrity, then reliability is improved, but processing speed deteriorates
Solution Approach 1:
The patent segments the data dictionary into two distinct tiers: a primary tier using probabilistic data structures (Bloom filters, Cuckoo filters) for rapid filtering, and a secondary tier using collision-resistant hashing for verification. This segmentation allows the system to use fast probabilistic methods for initial duplicate detection while reserving slower but more reliable cryptographic hashing only for verification of suspected duplicates, thereby resolving the contradiction between reliability and processing speed.
2Productivity
If extensive RAM resources are allocated for data dictionary, then deduplication capability is improved, but device complexity increases
Solution Approach 1:
The patent changes the fundamental parameters of the data dictionary by transitioning from traditional single-structure designs to a two-tiered architecture with different memory allocation strategies. The primary tier uses probabilistic structures optimized for speed with configurable false positive rates, while the secondary tier uses cryptographic structures for verification. This parameter change allows the system to achieve high deduplication capability without proportionally increasing complexity, as each tier is optimized for its specific function.
3Ease of manufacture
If traditional data dictionary structures are used, then implementation simplicity is maintained, but write performance deteriorates
Solution Approach 1:
The patent segments the data dictionary into two operational tiers with distinct access patterns and data structures. The primary tier uses probabilistic filtering for rapid write-time duplicate detection, significantly improving write performance. The secondary tier handles verification. This segmentation maintains implementation feasibility by building upon well-understood cryptographic hashing while adding layered optimization, rather than requiring complete system redesign.
Solution Approach 2:
The patent introduces probabilistic data structures (Bloom filters, Cuckoo filters) as intermediary layers between the incoming data and the final cryptographic verification. These intermediaries perform preliminary filtering to identify potential duplicates before expensive cryptographic operations are performed, thereby improving write performance while maintaining the reliability guarantee of collision-resistant hashing for final verification.
Data Source
AI summary
A data deduplication process maintains a data dictionary including a storage tablet and a secondary index containing data indicative of previously received data blocks. The tablet includes hashes of previous data blocks and the index includes one or more cuckoo filters storing fingerprints derived from block hashes of previous data blocks. When a new data block arrives, its block hash and fingerprint are generated. The storage tablet is queried with the block hash and the secondary index is queried with the fingerprint. If the dictionary contains no matching block hash or fingerprint, the new data block is stored in its entirety. If the dictionary contains a matching block hash or fingerprint, the new data block may be a duplicate data block that can be deduplicated by storing a reference to the previous data block instead of storing the new data block in its entirety.