Bitstream Compression with Coalesced Buffers and Delayed Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing demand for data storage and transfer over the Internet, coupled with limited bandwidth, necessitates more efficient bitstream compression and processing techniques to optimize bandwidth usage without compromising data integrity.
Innovation Solution
A novel technique that accelerates Longest Prefix Matching (LPM) algorithms for sequential data compression using a sliding window-based approach, optimizing hardware implementation for scalable and flexible compression applications, particularly with the DEFLATE algorithm, and incorporating token coalescing and efficient bitstream writing methods to enhance compression efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If lossless compression techniques are used to optimize bandwidth usage, then data transfer efficiency is improved, but computational cost increases
Solution Approach 1:
The patent pre-calculates and stores position information (offset and length) for all potential matches in the sliding window during a preprocessing phase. This preliminary action allows the main compression algorithm to quickly retrieve match information without performing expensive search operations during compression, thereby reducing computational cost while maintaining compression efficiency.
Solution Approach 2:
The patent creates a copy of the sliding window data in a separate data structure (match table) that organizes information by position rather than by sequence. This copying transforms the data layout to enable O(1) lookup of match information, avoiding the need to search through the entire sliding window during compression operations.
2Quantity of substance
If compression algorithms are implemented to reduce data size, then bandwidth requirements are reduced, but processing time increases
Solution Approach 1:
The patent performs preliminary organization of match data by pre-calculating offset and length values for all possible matches and storing them in an optimized data structure. This preprocessing eliminates the need for time-consuming search operations during the actual compression process, significantly reducing processing time while maintaining compression ratios.
Solution Approach 2:
The patent divides the compression process into distinct phases: a preprocessing phase that prepares match information, and a compression phase that uses the pre-prepared information. This segmentation allows computationally intensive operations to be performed once during preprocessing, while the actual compression runs efficiently using the pre-computed data.
3Reliability
If sliding window-based compression is used to improve compression ratio, then data integrity is maintained, but hardware complexity increases
Solution Approach 1:
The patent implements a simplified hardware structure by copying essential match information (offset and length) into a dedicated match table data structure. This copying approach replaces complex sliding window management hardware with a simpler table lookup mechanism, reducing hardware complexity while maintaining the ability to accurately track and verify match positions for data integrity.
Solution Approach 2:
The patent introduces a match table as an intermediary data structure between the sliding window buffer and the compression output. This intermediary pre-organizes match information and provides a simplified interface for the compression algorithm, reducing the complexity of direct sliding window management while ensuring accurate match detection and data integrity.
Data Source
AI summary
Methods and apparatus for processing bitstreams and byte streams. According to one aspect, bitstream data is compressed using coalesced string match tokens with delayed matching. A matcher is employed to perform search string match operations using a shortened maximum string length search criteria, resulting in generation of a token stream having <len, distance> data and literal data. A distance match operation is performed on sequentially adjacent tokens to determine if they contain the same distance data. If they do, the len values of the tokens are added through use of a coalesce buffer. Upon detection of a distance non-match, a final coalesced length of a matching string is calculated and output along with the prior matching distance as a coalesced token. Also disclosed is a scheme for writing variable-length tokens into a bitstream under which token data is input into a bit accumulator and written to memory (or cache to be subsequently written to memory) as each token is processed in a manner that eliminates branch mispredict operations associated with detecting whether the bit accumulator is full or close to full.