Pipelined Hardware Compression Engine With Parallel Hash Lookup
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing software-based inline lossless data compression in enterprise storage systems faces challenges in achieving high compression speed and low latency, as software compression algorithms do not map well to hardware implementations on FPGA or ASIC, leading to inefficiencies in CPU load and compression ratio.
Innovation Solution
A fully pipelined hardware compression engine design that utilizes efficient hash tables for concurrent lookup and update accesses, eliminating dead cycles and reducing false hash table hits, ensuring both high compression speed and low latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If software compression algorithms are used, then compression functionality is achieved, but compression speed and throughput are limited
Solution Approach 1:
The patent replaces software-based compression algorithms with a hardware compression engine implemented on FPGA or ASIC. This substitution of mechanical/computational system (software executing on CPU) with a dedicated hardware system enables parallel processing and eliminates the bottleneck of general-purpose CPU execution, thereby achieving higher compression speeds and throughput while reducing CPU load.
Solution Approach 2:
The patent introduces a dedicated hardware compression engine as an intermediary component between the storage system and data flow. This specialized hardware module acts as a mediator that performs compression operations in parallel, offloading the CPU and enabling high-speed compression without requiring software algorithm execution on the main processor.
2Productivity
If traditional hash tables are used in hardware, then lookup functionality is achieved, but dead cycles and false hits reduce efficiency
Solution Approach 1:
The patent segments the hash table into multiple independent hash tables, each handling a portion of the lookup operations. This segmentation allows concurrent access to different hash tables without interference, eliminating dead cycles that occur when a single hash table must be accessed sequentially. It also reduces false hits by distributing data across multiple tables with different hash functions.
Solution Approach 2:
The patent performs preliminary hashing and data organization into multiple hash tables before the actual compression operation. This preliminary action ensures that during compression, lookups can proceed in parallel without waiting for hash table construction or initialization, thereby reducing latency and eliminating dead cycles.
3Productivity
If more hardware resources are allocated for compression, then compression speed increases, but hardware resource utilization becomes inefficient
Solution Approach 1:
The patent designs the hardware compression engine with universal components that can handle multiple compression operations simultaneously. The same hardware resources (logic elements, memory, interconnects) are reused across different compression tasks and data streams, achieving high compression speeds without proportionally increasing hardware resource allocation. This multi-functionality ensures efficient resource utilization.
Solution Approach 2:
The patent merges multiple compression functions and operations into a single integrated hardware compression engine. By combining hash table lookups, compression algorithm execution, and output generation into one unified hardware structure, the system achieves high throughput while avoiding the overhead of multiple separate hardware components, thereby improving hardware resource efficiency.
Data Source
AI summary
A hardware compression system is provided. The system includes a hardware pipeline having a plurality of stages arranged to receive pre-compression data into a data buffer, populate a first hash table and a second hash table, supply the pre-compression data to a hash lookup module to access the first hash table and the second hash table in parallel, supply a string match module with results from the hash lookup module so that the string match module compares pre-compression data from multiple locations in the data buffer in parallel, supply a match merge module with results from the string match module so that the match merge module generates literals and metadata for compression data, and supply an output encoding module with results from the match merge module so that the output encoding module encodes the compression data.