GPU-Accelerated Data Deduplication with Bloom Filter
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data deduplication techniques are inefficient due to serial processing of hash algorithms, which can impact CPU utilization and I/O latency, and do not effectively utilize parallel processing capabilities, leading to suboptimal performance in data storage systems.
Innovation Solution
The method employs a GPU domain for parallel processing of data deduplication hash values using a Bloom filter to determine duplicate data portions, offloading CPU-intensive tasks and utilizing the GPU for simultaneous calculation of multiple hash values, while the Bloom filter provides probabilistic indications of data existence in a deduplication store.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If serial processing of hash algorithms is used for data deduplication, then CPU utilization is maintained, but processing efficiency and I/O latency are degraded
Solution Approach 1:
The patent segments the data deduplication process by dividing incoming data into fixed-size chunks and processing each chunk independently through parallel hash calculation on the GPU. This segmentation enables concurrent processing of multiple data portions simultaneously, resolving the contradiction between processing efficiency and I/O latency by eliminating the serial processing bottleneck while maintaining manageable task units.
Solution Approach 2:
The patent substitutes the traditional CPU-based serial processing mechanism with a GPU-based parallel processing mechanism. By leveraging the GPU's architecture designed for simultaneous execution of multiple threads, the system achieves significant throughput improvement for hash calculation, directly addressing the efficiency-latency contradiction without requiring changes to the fundamental deduplication logic.
2Productivity
If CPU is used for hash calculation, then processing is simple to implement, but parallel processing capabilities are underutilized
Solution Approach 1:
The patent applies universality by using the GPU for multiple functions: hash calculation, Bloom filter operations, and data chunk processing. This multi-functional approach maximizes the utilization of parallel processing capabilities while consolidating operations that would otherwise require separate processing units, thereby improving productivity without proportionally increasing system complexity.
Solution Approach 2:
The patent introduces a hybrid processing architecture where the CPU acts as an intermediary that prepares data for GPU processing and handles control logic, while the GPU performs computationally intensive hash calculations. This intermediary arrangement allows the system to leverage GPU parallelism without completely redesigning the control flow, balancing productivity improvement with manageable complexity.
3Reliability
If full hash verification is performed for every data portion, then deduplication accuracy is ensured, but processing overhead increases
Solution Approach 1:
The patent implements partial verification by using the Bloom filter to perform a quick probabilistic check before full hash verification. This partial action filters out many non-duplicate chunks that would otherwise require complete verification, thereby maintaining deduplication accuracy for true duplicates while significantly reducing processing overhead and improving throughput.
Solution Approach 2:
The patent performs preliminary filtering using the Bloom filter structure before committing to full hash verification. By pre-processing data chunks through the Bloom filter, the system identifies potential duplicates that warrant further verification while quickly dismissing non-duplicates, thus preserving reliability for actual duplicates while enhancing overall processing throughput.
Data Source
AI summary
Techniques for data deduplication may include: receiving write operations that write first data; partitioning the first data into a plurality of data portions; generating, using a first hash function, a plurality of data deduplication hash values for the plurality of data portions, wherein a first data deduplication hash value of the plurality of data deduplication hash values is produced by said generating for a first of the plurality of data portions; performing first processing using a Bloom filter to determine whether the first data deduplication hash value has a corresponding first entry in a data store of deduplication hash values; and responsive to the first processing determining the first data deduplication hash value does not have the corresponding first entry in the data store of deduplication hash values, performing second processing, said second processing including adding the corresponding first entry in the data store of deduplication hash values.