Parallel Hash Compression Using Hash Chains for Stable Throughput
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression systems, particularly those using content addressable memory (CAM) and hash-based architectures, face scalability issues with larger history sizes and variable throughput, leading to inefficiencies in processing and resource utilization, especially in FPGA implementations.
Innovation Solution
A parallel hash-based data compression method that generates hash values for incoming data streams, looks up potential match locations in a linked list of hash chains, and checks these locations in history buffer RAM for valid matches, allowing for simultaneous evaluation of multiple matches and efficient resource allocation through varying RAM sizes and lazy matching techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If CAM-based architecture is used to search for optimal matches among historical data, then every possible match can be tracked to find the longest match, but it doesn't scale well with larger history sizes and causes routing problems
Solution Approach 1:
The patent divides the history buffer into multiple segments and uses separate hash chains for each segment. This segmentation allows the system to handle large history sizes by distributing the search across multiple smaller, manageable segments, avoiding the routing complexity of a single large CAM structure while maintaining comprehensive match search capability.
Solution Approach 2:
The patent introduces hash chains as an intermediary data structure between the input data and the history buffer. Instead of directly searching the entire history buffer, the hash chains provide a structured intermediate layer that organizes potential match locations, simplifying the search process and reducing routing complexity while preserving match accuracy.
2Manufacturing precision
If hash-based compression architecture evaluates several potential matches to determine the best match, then compression quality is improved, but the throughput becomes variable based on the number of potential matches searched
Solution Approach 1:
The patent performs preliminary organization of match locations using hash chains before the actual match evaluation. By pre-organizing potential match locations in hash chains, the system reduces the number of matches that need to be evaluated in real-time, ensuring high compression quality while maintaining consistent throughput by limiting the evaluation to a manageable subset of pre-organized candidates.
Solution Approach 2:
The patent evaluates a limited number of potential matches from the hash chain rather than exhaustively searching all possible matches. This partial evaluation approach maintains acceptable compression quality by checking sufficient candidates without the need to evaluate every possible match, thereby ensuring stable throughput while achieving good compression results.
3Loss of substance
If LZ compression searches for repeated strings in a stream of data, then compression ratio is improved, but the search is time consuming and resource-intensive in terms of hardware needed
Solution Approach 1:
The patent replaces the traditional mechanical string searching approach with a hash-based computational system. Instead of sequentially comparing strings in the history buffer, the system uses hash functions to directly compute and locate potential match positions, dramatically reducing search time and hardware resources while maintaining the ability to identify and eliminate data redundancy for compression.
Data Source
AI summary
A method of parallel hash-based data compression comprising generating a hash value based on one or more original input data bytes of an incoming data stream using a processor, looking up, by the processor, a hash head comprising a first potential match location identified by the hash value and looking up, by the processor, at least one additional potential match location in a hash chain that comprises a linked list of potential match locations starting with a location indexed by the hash head. The method further comprises checking, by the processor, the at least one potential match location of the hash chain for one or more valid matches in a history buffer random access memory (RAM) and outputting, by the processor, a length and distance pair if a valid match is detected or the original input data byte if a match is undetected.