SDXI Hardware Accelerator for Inline Data Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data deduplication processes in information handling systems are computationally intensive and often performed offline, as they require significant processing resources and can be cumbersome due to large hash tables, making in-line deduplication difficult.
Innovation Solution
A Smart Data Accelerator Interface (SDXI) device that includes a hash function, a Bloom filter, and a data comparator to identify and eliminate duplicate data blocks independently of the processor, using hash functions to determine matches and a Bloom filter to quickly identify potential duplicates, with a data comparator confirming identical blocks and storing pointers for efficient deduplication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data deduplication is performed using traditional hash tables, then duplicate data blocks can be identified, but the process becomes computationally intensive and requires significant processing resources
Solution Approach 1:
The patent segments the deduplication process into multiple stages: first using a Bloom filter for rapid initial screening of potential duplicates, then using a hash table for definitive verification. This segmentation allows the system to quickly filter out non-duplicates without expensive hash computations, while maintaining accurate duplicate detection for the remaining candidates.
Solution Approach 2:
The patent applies different data structures with different qualities to different parts of the deduplication process. The Bloom filter provides a fast, memory-efficient first layer of filtering, while the hash table provides accurate but more resource-intensive verification. Each data structure is optimized for its specific role in the overall deduplication workflow.
2Loss of substance
If in-line data deduplication is implemented, then storage efficiency improves, but processing resources and time are consumed during data transfer
Solution Approach 1:
The patent performs preliminary duplicate detection using the Bloom filter before committing resources to full hash verification. By pre-screening data blocks with the fast Bloom filter during the data transfer process, the system can identify and skip expensive hash computations for non-duplicates, enabling in-line deduplication with minimal impact on transfer performance.
3Productivity
If a Bloom filter is used for quick duplicate identification, then processing speed improves, but false positives may occur requiring additional verification
Solution Approach 1:
The patent uses the Bloom filter as an intermediary screening mechanism that quickly identifies potential duplicates without making final determinations. The Bloom filter acts as a gatekeeper that directs only promising candidates to the hash table verification stage, reducing the overall verification burden while maintaining accurate final duplicate identification.
Data Source
AI summary
A hardware accelerator device detects duplicate data blocks independently from a processor of an information handling system. The device includes an interface, a hash function, a bloom filter, and a data comparator. The interface receives data blocks. The hash function determines a hash of a received data block. The bloom filter includes comparison hashes, each associated with a comparison data block. The bloom filter determines if the hash matches a comparison hash. The data comparator directs the interface to receive the comparison block when the hash matches the comparison hash, and compares the received data block with the comparison data block to determine if they are identical. The device stores a pointer when the data comparison block determines that the blocks are identical.