Hash-Based Compression Processor for High-Throughput LZ Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing hardware implementations of data compression algorithms like GZIP, based on DEFLATE and Huffman coding, face complexity in achieving high bandwidth due to the need for a search history buffer and compare length functions, which can be inefficient in terms of processing and resources.
Innovation Solution
The development of an improved hashing method that generates a hash chain to quickly index data sets using a hash table and hash chain, allowing for efficient compression by suppressing hash generation during high throughput requirements and using multiple memory banks with FIFOs to manage collisions, thereby maintaining high compression rates with minimal overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a search history buffer and compare length function are used to implement compression, then compression capability is improved, but device complexity and processing overhead increase
Solution Approach 1:
The patent extracts the essential compression functionality from the complex DEFLATE algorithm by implementing only the LZ77 component with a simplified hash-based search mechanism. This removes unnecessary complexity while retaining core compression capability, achieving over 90% compression rates with reduced hardware overhead.
Solution Approach 2:
The patent changes the search parameter from comparing full strings to comparing hash values, which transforms the complexity from O(n*m) to O(1) for the initial search phase. This parameter change in the search strategy dramatically reduces processing complexity while maintaining compression effectiveness.
2Speed
If a hash function is used to map variable length strings to fixed values for comparison, then search speed is improved, but hash collisions increase
Solution Approach 1:
The patent introduces a two-stage verification process where the hash value acts as an intermediary filter. First, hash values are compared for quick filtering, and only matching hash values proceed to full string comparison. This intermediary step dramatically reduces false positives from hash collisions while maintaining high search speed.
Solution Approach 2:
The patent segments the search process into two independent stages: hash value comparison and full string comparison. This segmentation allows the system to use the fast but imperfect hash function for initial filtering, then apply the more reliable but slower full comparison only when necessary, optimizing both speed and reliability.
3Productivity
If multiple memory banks with FIFOs are used to manage hash collisions, then compression throughput is improved, but device complexity increases
Solution Approach 1:
The patent segments the single hash table into multiple independent memory banks, each with its own FIFO queue for handling collisions. This segmentation allows parallel processing of hash entries across different banks, increasing throughput while keeping each individual bank relatively simple in structure.
Solution Approach 2:
The patent implements preliminary hashing to determine the target bank before data arrives, allowing each memory bank to be pre-configured and ready for immediate processing. This preliminary action eliminates dynamic allocation overhead during compression operations, improving throughput while maintaining manageable complexity.
Data Source
AI summary
An input file is processed according to hash algorithm that references sets of literals to preceding sets of literals to facilitate copy-offset command generation. Preceding instances are identified by generating a hash of the literal set and looking up a corresponding entry in a hash table. The hash table may be accessed by placing look-up requests in a FIFO buffer. When the FIFO buffer is full, generation of the hash chain is suspended until it is no longer full. When repeated literals are found, generation of the hash chain is likewise suspended. The hash chain is used to generate a command file, such as according to the LZ algorithm. Runs of consecutive literals are replaced by a run-length command. The command file may then be encoded using Huffman encoding.