Deduplication Pipeline Latency Reduction via QAT Offloading
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional deduplication pipelines face bottlenecks due to CPU processing limitations and NVRAM performance, leading to increased latency and compromised compression ratios, as they struggle to fully utilize new hardware technologies like QAT and PMEM.
Innovation Solution
Offloading compression to a co-processor like QAT and reorganizing the pipeline to store uncompressed data in PMEM, allowing parallel processing and direct access for faster write operations, thus separating compression latency from the overall pipeline latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If higher compression ratios are used to generate less data to be saved, then storage utilization is improved, but processing time increases due to more compute operations
Solution Approach 1:
The patent extracts the compression function from the main CPU processing pipeline and implements it as a separate co-processor (QAT - Quick Assist Technology). This allows compression operations to be performed in parallel with other deduplication tasks, eliminating the bottleneck where high compression ratios caused processing delays in the main pipeline.
Solution Approach 2:
The patent implements preliminary action by performing deduplication filtering and identifying duplicate data blocks before compression. The pipeline first filters data to identify unique blocks, then compresses only those unique blocks in parallel, rather than compressing all data first. This preliminary filtering reduces the total amount of data requiring compression while maintaining high compression ratios on the unique portions.
2Quantity of substance
If CPU is used for compression operations to achieve high compression ratios, then storage efficiency is improved, but CPU becomes a bottleneck increasing overall latency
Solution Approach 1:
The patent extracts the compression function from the main CPU processing pipeline and implements it as a separate co-processor (QAT - Quick Assist Technology). This allows compression operations to be performed in parallel with other deduplication tasks, eliminating the bottleneck where high compression ratios caused processing delays in the main pipeline.
Solution Approach 2:
The patent introduces a co-processor (QAT) as an intermediary between the CPU and storage system specifically for compression operations. This intermediary handles the computationally intensive compression tasks, allowing the main CPU pipeline to continue processing other deduplication operations without waiting for compression to complete, thus maintaining high throughput while achieving efficient compression.
3Reliability
If NVRAM is used for buffering to maintain data integrity, then reliability is improved, but performance is limited due to bottleneck constraints
Solution Approach 1:
The patent introduces persistent memory as an intermediary buffering layer between the fast storage system and the slower NVRAM. This persistent memory buffer absorbs the speed difference, allowing data to be written quickly to persistent memory while maintaining the reliability guarantee of eventual NVRAM storage. The system can acknowledge writes faster without sacrificing data integrity.
Solution Approach 2:
The patent segments the buffering function across multiple layers: persistent memory for high-speed intermediate buffering and NVRAM for reliable final storage. This segmentation allows each layer to optimize for its specific function - persistent memory for speed and NVRAM for reliability - while working together to provide both high performance and data integrity.
Data Source
AI summary
A deduplication pipeline method to enable shorter overall latency, servicing of multiple calls in parallel, and implementing higher data compression ratio. The method includes receiving user data for storage, performing deduplication operation on the data to obtain non-duplicative data, buffering the non-duplicative data in persistent memory, and accepting next user data for deduplication processing. In parallel to receiving the next user data, operating a co-processor to asynchronously compressing the data stored in the persistent memory and storing the compressed data in RAID.