Double-Pass Lempel-Ziv Compression With Adaptive Encoding Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data compression techniques face challenges in achieving high compression ratios while maintaining efficient processing speed, particularly in lossless data compression where redundancy is high, and there is a need for adaptive methods that can select optimal encoding strategies based on data characteristics.
Innovation Solution
The implementation of a system that uses Lempel-Ziv modeling with adaptive encoding strategies, including single-pass and double-pass modeling, hash tables, and Huffman trees, to identify and encode byte sequences, and automatically selects the best encoding type based on parsing quality and data characteristics, utilizing hash tables, Markov chain compression, and custom prefix generation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional data compression techniques are used, then processing speed is maintained, but compression ratios are insufficient for high-redundancy data
Solution Approach 1:
The patent divides the input data stream into multiple byte sequences and processes them through separate hashing and matching stages. The compression process is segmented into identification, hashing, and encoding phases, allowing parallel processing of different data portions while maintaining high compression ratios through comprehensive redundancy detection.
Solution Approach 2:
The patent performs preliminary hashing of byte sequences before the main compression process. By pre-computing hash values and storing them in hash tables during an initial pass, the system prepares data structures that enable rapid matching and compression decision-making in subsequent processing stages.
2Loss of substance
If adaptive encoding strategies are implemented, then compression ratios improve, but device complexity increases
Solution Approach 1:
The patent implements dynamic selection of encoding strategies by evaluating multiple compression approaches (different hashing methods, encoding schemes) and automatically choosing the optimal strategy based on data characteristics. The system adapts its complexity level to match the redundancy patterns in the input data, using more complex methods only when they provide measurable compression benefits.
Solution Approach 2:
The patent changes encoding parameters such as hash table size, byte sequence length, and encoding method based on the observed data characteristics. By dynamically adjusting these parameters rather than using fixed complex algorithms, the system achieves high compression ratios for suitable data types while maintaining simplicity for data where basic compression suffices.
3Loss of substance
If hash tables and Markov chain compression are used, then compression performance improves, but memory requirements increase
Solution Approach 1:
The patent implements partial hashing where only critical portions of byte sequences are hashed and stored in hash tables, rather than hashing entire sequences. The Markov chain compression is applied selectively to identify and compress only the most redundant patterns in the data, using memory resources proportionally to the actual redundancy present in the input data rather than allocating fixed large memory buffers.
Data Source
AI summary
A method includes receiving an input data stream at a processor, and for each byte sequence from a plurality of byte sequences of the input data stream, a hash is generated and compared to a hash table to determine whether a match exists. If a match exists, that byte sequence is incrementally expanded to include one or more additional adjacent bytes from the input data stream, to produce multiple expanded byte sequences. Each of the expanded byte sequences is compared to the hash table to identify a maximum-length matched byte sequence from a set that includes the byte sequence and the plurality of expanded byte sequences. A representation of the maximum-length matched byte sequence is stored in the memory. If a match does not exist, a representation of that byte sequence is stored as a byte sequence literal in the memory.