Pipelined Data Compression Using Dictionary Match Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data compression methods, such as Deflate, face limitations in achieving high bandwidth compression for encoded data streams, particularly in efficiently processing repeated data patterns across multiple cycles, leading to suboptimal throughput and latency.
Innovation Solution
A data processing system that combines current and next cycle data to form frames, uses dictionary memories for lookup and match selection, and encodes data using Huffman coding to achieve high bandwidth compression by identifying and encoding matches across multiple substrings, thereby optimizing data coverage and reducing latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional Deflate compression is used, then lossless compression is achieved, but throughput is limited and latency increases
Solution Approach 1:
The patent segments the compression process into distinct pipeline stages: dictionary lookup stage, match selection stage, and encoding stage. Each stage processes data independently and passes results to the next stage, enabling parallel processing and eliminating sequential bottlenecks. This segmentation allows multiple operations to occur simultaneously, dramatically improving throughput while maintaining compression quality.
Solution Approach 2:
The patent performs preliminary actions by pre-building and maintaining a dictionary of previously seen data patterns in parallel with the main compression process. The dictionary lookup stage proactively searches for matches using hash tables before the encoding stage needs to process the data, reducing the critical path delay and enabling faster compression decisions.
2Loss of substance
If comprehensive match searching is performed, then compression ratio improves, but processing complexity increases
Solution Approach 1:
The patent introduces intermediary data structures including hash tables for rapid dictionary lookup and bit buffers for efficient match representation. These intermediaries simplify the matching process by transforming complex pattern recognition into straightforward table lookups and bitwise operations, reducing processing complexity while enabling comprehensive match searching across the data stream.
Solution Approach 2:
The patent replaces mechanical sequential searching with algorithmic hash-based lookup mechanisms. Instead of linearly scanning through the dictionary to find matches, the system uses hash functions to directly compute memory addresses, reducing the time complexity from O(n) to O(1) for dictionary operations and enabling exhaustive match searching without proportionally increasing processing complexity.
3Productivity
If data is processed in larger frames, then compression efficiency improves, but memory requirements and processing overhead increase
Solution Approach 1:
The patent implements dynamic frame processing where the effective frame size adapts based on the data characteristics and available resources. The system dynamically adjusts the balance between gathering sufficient data for efficient compression and limiting frame size to control memory usage, allowing optimal compression efficiency without excessive memory requirements through adaptive parameter adjustment during processing.
Data Source
AI summary
A mechanism is provided in a data processing system for pipelined compression of multi-byte frames. The mechanism combines a current cycle of data in an input data stream with at least a portion of a next cycle of data in the input data stream to form a frame of data. The mechanism identifies a plurality of matches in a plurality of dictionary memories. Each match matches a portion of a given substring in the frame of data. The mechanism identifies a subset of matches from the plurality of matches that provides a best coverage of the current cycle of data. The mechanism encodes the frame of data into an encoded output data stream.