Hybrid DEFLATE Compression Split Between CPU and Hardware
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing lossless data compression algorithms, such as DEFLATE, face low throughput when implemented on CPUs, and offloading the entire algorithm to hardware accelerators leads to interface bandwidth limitations and high implementation costs due to the need for raw data transfer and significant silicon resource consumption.
Innovation Solution
A hybrid approach where CPU performs byte-oriented operations like matching-bytes search and content analysis, while hardware accelerators handle bit-oriented operations like search result re-coding and Huffman encoding, optimizing data transfer and resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the entire DEFLATE algorithm is off-loaded to a hardware accelerator, then compression throughput is improved, but interface bandwidth becomes the limiting factor and implementation cost increases
Solution Approach 1:
The DEFLATE algorithm is divided into two segments: byte-oriented operations (matching-bytes search, byte-oriented search result coding, content analysis) are performed on the CPU, while bit-oriented operations (search result re-coding, table construction, encoding) are off-loaded to the hardware accelerator. This segmentation allows the system to leverage the high interface bandwidth available for smaller data transfers while still benefiting from hardware acceleration for the most computationally intensive operations.
2Productivity
If the entire DEFLATE algorithm is off-loaded to a hardware accelerator, then compression throughput is improved, but silicon resource consumption increases leading to higher implementation cost
Solution Approach 1:
The byte-oriented operations are extracted from the hardware accelerator and performed on the CPU instead. This reduces the complexity and silicon resource requirements of the accelerator while maintaining the throughput benefits of hardware acceleration for the remaining bit-oriented operations.
3Ease of manufacture
If DEFLATE is implemented on CPU, then implementation cost is reduced, but compression throughput becomes significantly inadequate
Solution Approach 1:
The algorithm is segmented between CPU and hardware accelerator based on operation type. The CPU handles byte-oriented operations while the accelerator handles bit-oriented operations, combining the cost-effectiveness of software implementation with the high throughput of hardware acceleration.
4Manufacturing precision
If raw data is transferred to hardware accelerator for compression, then compression ratio is maintained, but interface bandwidth limits achievable throughput
Solution Approach 1:
The data processing is segmented into two stages: CPU performs initial compression on raw data to generate intermediate compressed data, then transfers only this reduced data to the accelerator for final compression. This reduces the transfer volume and maximizes the utilization of interface bandwidth.
Data Source
AI summary
A system, method and product for providing data compression and decompression. A method is disclosed that includes: utilizing a CPU to perform a matching-bytes search, byte-oriented search result coding, and content analysis on a set of raw data to generate a set of initially compressed data; forwarding the set of initially compressed data from the CPU to a hardware accelerator; utilizing the hardware accelerator to perform search result re-coding, table construction, and encoding to generate a set of further compressed data; and forwarding the set of further compressed data back to the CPU.