Hash-Based Compression Matching for Larger History Buffers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data compression algorithms, such as the Lempel-Ziv Stac algorithm, rely on pattern matching to reduce data transmission bandwidth, but they are limited by the need for precise matching between transmitter and receiver history buffers, leading to inefficiencies in compressing data with varying redundancy levels.
Innovation Solution
The method involves generating a hash table with pointers to historical data, allowing for doubleword and all-to-all comparisons to increase the chances of pattern matching, thereby optimizing data compression by using additional data from the history and its associated pointers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional pattern matching algorithms are used to search for repeating data patterns in the history buffer, then the compression algorithm can identify and replace repetitive data, but the search process becomes time-consuming and reduces compression efficiency
Solution Approach 1:
The patent pre-processes the history buffer by creating a hash table that stores hash values of historical data segments along with their positions. This preliminary action allows the compression algorithm to quickly retrieve potential matches without performing exhaustive sequential searches, thereby maintaining pattern matching accuracy while significantly improving compression efficiency.
Solution Approach 2:
The patent introduces a hash table as an intermediary data structure between the input data and the history buffer. The hash table serves as a mediator that maps hash values to historical data positions, enabling fast indirect access to potential pattern matches without direct comparison of entire data segments, thus resolving the contradiction between matching reliability and compression productivity.
2Productivity
If the history buffer size is increased to improve compression ratios for highly redundant data, then more patterns can be matched, but the memory requirements and buffer management complexity increase
Solution Approach 1:
The patent segments the history buffer into manageable units and organizes them in a hash table structure. Instead of treating the entire history buffer as a single large block that requires complex management, it divides the buffer into smaller segments that can be independently hashed and stored, reducing buffer management complexity while still allowing access to a large effective history for improved compression ratios.
Solution Approach 2:
The patent transitions from a one-dimensional sequential history buffer to a two-dimensional hash table structure where data is organized by hash value indices. This dimensional change allows efficient access to historical patterns without requiring linear search through the entire buffer, enabling larger effective buffer sizes with manageable complexity through the hash-based indexing system.
Data Source
AI summary
Systems and methods are provided for data comparisons in a compression algorithm to optimize the compression of data. An exemplary method includes receiving input data. The method further includes generating a hash table address for the input data. The method further includes obtaining a pointer associated with the hash table address. The method further includes reading a data library with the pointer to obtain an associated piece of historical data. The method further includes comparing the historical data to the input data to determine a match.