Hardware Data Compression with Dynamic-Prime Huffman Tables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data compression methods using dynamic Huffman code tables in DEFLATE-style compression are inefficient due to sequential processing, which delays Huffman encoding until histograms are fully generated, and sorting of histograms is time-consuming, adding to the overall compression time.
Innovation Solution
A hardware data compressor that constructs dynamic-prime Huffman code tables after scanning an initial fraction of the input block and sorts the histogram incrementally concurrently with scanning, allowing parallel processing of LZ77 compression and Huffman coding, thereby hiding sorting and Huffman coding times within the LZ77 compression time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If dynamic Huffman code tables are constructed after full histogram generation, then compression accuracy is improved, but compression time increases
Solution Approach 1:
The patent applies preliminary action by constructing Huffman code tables after scanning only an initial fraction (e.g., first 1024 bytes) of the input block, rather than waiting for complete histogram generation. This early construction allows the main compression loop to proceed in parallel, significantly reducing total compression time while maintaining acceptable compression accuracy through subsequent table updates.
Solution Approach 2:
The patent implements dynamics by transitioning from static Huffman tables to dynamically updated tables during compression. The system starts with tables constructed from initial histogram data, then continuously updates them as more input data is processed, allowing the compression algorithm to adapt to the actual data distribution while maintaining high compression ratios.
2Manufacturing precision
If histogram sorting is performed sequentially, then sorting accuracy is improved, but compression speed decreases
Solution Approach 1:
The patent applies preliminary action by performing histogram sorting on only the initial fraction of scanned data before entering the main compression loop. This preliminary sorting provides sufficiently accurate Huffman tables to start compression, while the full sorting will be completed or updated later, thus avoiding the bottleneck of complete sequential sorting.
Solution Approach 2:
The patent maintains continuity of useful action by overlapping the histogram sorting process with the LZ77 compression scanning. While the compressor scans and compresses data using initial Huffman tables, the histogram continues to be populated and sorted in the background, ensuring that useful compression work continues without interruption while sorting accuracy is progressively improved.
3Manufacturing precision
If Huffman coding is delayed until after LZ77 compression, then encoding accuracy is improved, but total processing time increases
Solution Approach 1:
The patent applies preliminary action by constructing initial Huffman code tables before the main LZ77 compression loop begins. These preliminary tables allow the encoding process to start immediately in parallel with compression, rather than waiting for complete histogram analysis, thus reducing total processing time while maintaining acceptable encoding accuracy.
Solution Approach 2:
The patent implements dynamics by making Huffman code tables dynamic and updateable during the compression process. The tables are initially constructed from a fraction of the input, then continuously updated as more data is scanned and histograms are refined, allowing encoding accuracy to improve over time without delaying the start of compression.
Data Source
Figure 1~2A
Figure 2B~3
Figure 4
AI summary
A hardware data compressor for compressing an input block of characters. A first hardware engine that, for an initial fraction of the input block of characters produces character string replacement back pointers and indicates characters not replaced by the back pointers, and generates occurrence frequencies of symbols associated with the produced back pointers and the non-replaced characters. A second hardware engine constructs a Huffman code table using the frequencies generated for the initial fraction of the input block. The first hardware engine, for the remainder of the input block beyond the initial fraction, produces character string replacement back pointers and indicates characters not replaced by the back pointers. A third hardware engine, for the entire input block of characters, Huffman encodes the symbols associated with the back pointers and the non-replaced characters using the Huffman code table constructed using the frequencies generated for the initial fraction of the input block.