Hardware Data Compression with Concurrent Huffman Symbol Sorting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data compression methods using dynamic Huffman code tables in DEFLATE-style compression are inefficient due to sequential processing, which delays the Huffman encoding step until histograms are fully generated, leading to increased compression time.
Innovation Solution
A hardware data compressor that constructs 'dynamic-prime' Huffman code tables after scanning only an initial portion of the input block, allowing parallel processing with LZ77 compression and incremental sorting of the symbol list, thereby hiding Huffman coding time behind LZ77 compression time and reducing overall compression time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If sequential processing is used to generate histograms and then construct Huffman code tables, then the compression process is simple to implement, but the compression time increases
Solution Approach 1:
The patent applies preliminary action by constructing 'dynamic-prime' Huffman code tables after scanning only an initial portion of the input block, before the complete histogram is available. This allows the Huffman encoding process to start earlier and proceed in parallel with the remaining LZ77 compression work, rather than waiting for sequential completion of histogram generation.
Solution Approach 2:
The patent implements continuity of useful action by maintaining parallel processing between LZ77 compression and Huffman code table construction. The sort engine continuously updates the sorted symbol list as new symbols are encountered during scanning, keeping the Huffman construction process ongoing rather than pausing for sequential completion.
2Measurement precision
If Huffman code tables are constructed after complete histogram generation, then accurate frequency data is available, but compression time increases
Solution Approach 1:
The patent applies partial action by using an incomplete histogram from only the initial portion of the input block to construct the dynamic-prime Huffman code tables. Rather than requiring complete frequency data from the entire input block, the system proceeds with partial information, accepting some approximation in exchange for significant time savings through parallel processing.
Solution Approach 2:
The system performs preliminary Huffman code table construction based on initial frequency observations, allowing the encoding process to begin before complete frequency data is available. This preliminary action is then refined as more data comes in during the parallel processing phases.
3Loss of time
If parallel processing is used to overlap Huffman coding with LZ77 compression, then compression time is reduced, but device complexity increases
Solution Approach 1:
The patent applies segmentation by dividing the compression system into distinct parallel components: an LZ77 compression engine and a separate Huffman code table construction engine with its own sort engine. This segmentation allows independent parallel operation of each component while maintaining overall system functionality.
Solution Approach 2:
The patent uses an intermediary sorted symbol list that serves as the interface between the LZ77 compression process and the Huffman encoding process. The sort engine acts as a mediator, continuously maintaining this list as input from the LZ77 engine, enabling the Huffman engine to operate in parallel without direct dependency on the completion of LZ77 compression.
Data Source
Figure 1~2A
Figure 2B~3
Figure 4
AI summary
A hardware data compressor includes a first hardware engine that scans an input block of characters to produce a stream of tokens, the stream of tokens comprising replacement back pointers to matched strings of characters of the input block and non-replaced characters of the input block. The hardware data compressor also incudes a second hardware engine that receives the stream of tokens and maintains a sorted list of symbols associated with the tokens. The hardware data compressor also incudes the second hardware engine concurrently maintains the sorted list of symbols by frequency of occurrence as the first hardware engine produces the tokens of the stream.