Hash-Chain Data Compression with Probability-Sorted String Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data compression methods using dynamic Huffman code tables in DEFLATE-style compression are inefficient due to sequential processing, which delays Huffman encoding until histograms are fully generated, leading to increased compression time.
Innovation Solution
A hardware data compressor that constructs 'dynamic-prime' Huffman code tables after scanning only an initial portion of the input block, allowing parallel processing with LZ77 compression and incremental sorting of symbol frequencies, thereby hiding Huffman coding time behind LZ77 compression time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If dynamic Huffman code tables are constructed after full histogram generation, then encoding accuracy is improved, but compression time increases
Solution Approach 1:
The patent applies preliminary action by constructing Huffman code tables after scanning only an initial portion of the input block rather than waiting for complete histogram generation. This allows the Huffman encoding to begin in parallel with the remaining LZ77 compression work, overlapping the two operations to reduce total compression time while maintaining acceptable encoding accuracy through incremental sorting updates.
2Device complexity
If sequential processing is used for LZ77 compression and Huffman encoding, then implementation simplicity is improved, but productivity decreases
Solution Approach 1:
The patent implements continuity of useful action by enabling parallel processing where Huffman code table construction and incremental sorting occur concurrently with LZ77 compression. The sort engine continuously updates symbol frequency rankings as tokens are generated, eliminating idle time and keeping all processing units continuously productive throughout the compression pipeline.
Solution Approach 2:
The patent applies segmentation by dividing the compression workflow into independent parallel components: an LZ77 compression engine that scans and generates tokens, a sort engine that maintains symbol frequency rankings, and a Huffman encoding engine that constructs code tables. These segmented components operate simultaneously with defined interfaces, improving throughput without sacrificing implementation clarity.
3Manufacturing precision
If complete symbol frequency sorting is performed before Huffman encoding, then code optimality is improved, but time consumption increases
Solution Approach 1:
The sort engine performs preliminary sorting actions incrementally as symbol frequencies are updated during token generation. Rather than waiting for complete frequency counts, the sort engine maintains current rankings and updates them progressively, allowing Huffman encoding to begin with sufficiently optimized code tables while continuing to refine sorts as more data becomes available.
Solution Approach 2:
The patent implements dynamics by using an incremental sorting mechanism that adapts to changing symbol frequencies in real-time. The sort engine dynamically updates symbol rankings as new tokens are processed, transitioning from static pre-computation to dynamic adaptive sorting that maintains code optimality while reducing total sorting time through parallel execution.
Data Source
Figure 1~2A
Figure 2B~3
Figure 4
AI summary
A hardware data compressor. A first hardware engine scans an input block of characters and uses a plurality of lists of nodes to produce back pointers to matching strings in the input block to compress the input block. Each node points to a character in the input block previously scanned and has an associated probability that a back pointer to a matching string that begins with the pointed-to character will be produced by the first hardware engine. A second hardware engine, for each list of nodes of the plurality of lists, sorts the list according to the probabilities of the nodes in the list so that higher probability nodes appear earlier in the list for use by the first hardware engine to search for matching strings during the scan of the input block of characters.