Scatter-Gather Decompression With Combined Checksum Verification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing decompression process with Intel QuickAssist Technology (QAT) hardware accelerator incurs overhead due to submitting compressed data in smaller packets, as it cannot generate a combined checksum for combined uncompressed data, requiring individual submissions and limiting optimal decompression performance.
Innovation Solution
Submit compressed data from multiple compression units in a single submission to the QAT hardware accelerator, calculate a combined checksum on the decompressed data, and verify it against stored individual checksums to ensure correct decompression, allowing for improved decompression performance by reducing the number of submissions and overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If compressed data is submitted in smaller packets to the QAT hardware accelerator, then the decompression process can maintain individual checksum verification for each packet, but the number of submissions increases and overhead increases, reducing decompression performance
Solution Approach 1:
The patent segments the checksum verification process by calculating individual checksums for each compressed data packet before submission. This allows each packet to be independently verified while still enabling batch submission of multiple packets, thus maintaining reliability without sacrificing the performance benefits of larger packet submissions.
Solution Approach 2:
The patent performs preliminary checksum calculation on each compressed data packet before submission to the QAT hardware accelerator. By pre-calculating and storing checksums for each packet, the system enables efficient batch processing while maintaining the ability to verify data integrity, thus resolving the contradiction between reliability and productivity.
2Productivity
If compressed data from multiple compression units is combined into a single submission, then the number of submissions decreases and overhead is reduced, but the existing system cannot generate a combined checksum for verification
Solution Approach 1:
The patent merges multiple individual checksums into a single combined checksum that represents the integrity of all decompressed data from multiple compression units. This combined checksum enables verification of batch-decompressed data, thus allowing single submission of multiple compression units while maintaining data verification capability.
Solution Approach 2:
The patent introduces a combined checksum as an intermediary verification mechanism that bridges the gap between batch processing and individual verification. The combined checksum serves as a mediator that allows the system to verify the integrity of collectively decompressed data without requiring individual verification of each packet, thus enabling efficient batch processing with maintained reliability.
3Productivity
If larger data packets are submitted to the QAT hardware accelerator, then the overhead per unit of data decreases and performance improves, but the system must handle more complex checksum verification for combined data
Solution Approach 1:
The patent creates a copy of the checksum verification mechanism that operates at the batch level rather than the individual packet level. By generating a combined checksum that represents the entire batch of decompressed data, the system avoids the complexity of verifying each individual packet while maintaining verification capability, thus reducing the effective complexity for large packet processing.
Data Source
AI summary
A method, apparatus, and system for decompressing data with a hardware compression/decompression accelerator is disclosed. The operations comprise: submitting compressed data from a plurality of stored compression units in a compression region to a hardware compression/decompression accelerator in a single submission for decompression, wherein each compression unit stores a checksum calculated based on corresponding uncompressed data; decompressing, at the hardware compression/decompression accelerator, the compressed data from the plurality of stored compression units, the decompressing generating combined decompressed data corresponding to the compressed data; calculating, at the hardware compression/decompression accelerator, a first combined checksum based on the combined decompressed data; calculating a second combined checksum based on individual checksums stored in the plurality of compression units; determining whether the first combined checksum matches the second combined checksum; and if the combined checksums match, forwarding the combined decompressed data to a storage device for storage as uncompressed data.