Adaptive Hash Compression for High-Throughput LZ Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing hardware implementations of data compression algorithms like GZIP, based on DEFLATE and Huffman coding, face challenges in achieving high throughput due to the complexity of search history buffer management and hash function implementations required for Lempel-Ziv compression.
Innovation Solution
The development of an adaptive rate compression hash processor that uses a hash function to generate a deterministic index for data compression, employing a hash table and hash chain to efficiently index data sets, and incorporates hash suppression mechanisms to manage collisions and maintain high throughput.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a hash function is implemented in hardware for Lempel-Ziv compression, then compression throughput is improved, but device complexity increases due to search history buffer management and hash table requirements
Solution Approach 1:
The hash table is divided into multiple segments or banks, allowing parallel hash calculations and reducing the complexity of managing a single large hash table. This segmentation enables the system to process multiple data streams simultaneously while maintaining manageable buffer sizes for each segment.
Solution Approach 2:
The patent introduces a hierarchical structure with hash tables organized in multiple levels or dimensions. Instead of a single flat hash table, the system uses a multi-dimensional approach where hash values map to different levels of the structure, reducing the complexity of direct address mapping while maintaining compression throughput.
2Measurement precision
If a large hash table is used to improve compression rate, then compression accuracy is improved, but memory requirements and device complexity increase
Solution Approach 1:
The hash table is segmented into multiple smaller banks or sections, each handling a portion of the total hash space. This allows the system to achieve high compression accuracy through collective coverage of all segments while keeping individual memory requirements for each segment manageable.
Solution Approach 2:
The hash table structure is made dynamic with adjustable parameters such as table size, number of banks, and entry depth. This allows the system to optimize the balance between compression accuracy and memory requirements based on specific application needs, rather than requiring a fixed large table for all cases.
3Device complexity
If hash suppression is implemented to manage collisions, then device complexity is reduced, but compression rate may be affected
Solution Approach 1:
Hash suppression is applied selectively rather than universally - only to specific collision cases where it provides benefit. The system identifies and applies suppression mechanisms to the most problematic collision scenarios while leaving other cases to handle naturally, maintaining compression rate while reducing overall device complexity.
Solution Approach 2:
The suppression mechanism uses adjustable parameters to control its behavior, allowing optimization between complexity reduction and compression rate maintenance. By dynamically adjusting suppression thresholds and criteria, the system can adapt to different data patterns and maintain high compression rates while keeping the device complexity manageable.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
An input file is processed according to hash algorithm that references sets of literals to preceding sets of literals to facilitate copy-offset command generation. Preceding instances are identified by generating a hash of the literal set and looking up a corresponding entry in a hash table. The hash table may be accessed by placing look-up requests in a FIFO buffer. When the FIFO buffer is full, generation of the hash chain is suspended until it is no longer full. When repeated literals are found, generation of the hash chain is likewise suspended. The hash chain is used to generate a command file, such as according to the LZ algorithm. Runs of consecutive literals are replaced by a run-length command. The command file may then be encoded using Huffman encoding.