Two-Stage Data Compression for Long- and Short-Range Redundancy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data compression techniques, such as Lempel-Ziv family compressors, are inadequate for exploiting long-range redundancies in internet traffic and cannot compress already compressed entities, leading to suboptimal performance and increased data size.
Innovation Solution
A two-stage compression system comprising a long range compressor front end that captures macro redundancies using a large history buffer and a short range compressor back end with grammar transform and adaptive arithmetic coding to exploit micro redundancies, allowing efficient compression of entities that have already been compressed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If Lempel-Ziv family compressors are used with small dictionary size, then device complexity is reduced, but compression effectiveness deteriorates due to inability to exploit long-range redundancies
Solution Approach 1:
The patent segments the compression task into two distinct stages: a long-range compressor that exploits redundancies across large data spans using a large dictionary, and a short-range compressor that handles local patterns. This segmentation allows each stage to be optimized independently, resolving the contradiction between complexity and effectiveness.
Solution Approach 2:
The patent introduces an intermediary component that manages the interaction between the long-range and short-range compressors. This intermediary coordinates the processing flow and data exchange between stages, enabling the system to achieve high compression effectiveness without requiring either single-stage compressor to be overly complex.
2Productivity
If large history buffer is used to capture long-range redundancies, then compression effectiveness is improved, but device complexity increases
Solution Approach 1:
The patent divides the compression function into two stages with different history buffer requirements. The long-range compressor uses a large history buffer to capture distant redundancies, while the short-range compressor uses a small buffer for local patterns. This segmentation allows the system to achieve high compression effectiveness without requiring a single compressor to maintain excessive complexity.
Solution Approach 2:
The patent transitions from a single-dimension approach (one compressor with large buffer) to a two-dimensional approach (multiple compressors with different buffer sizes working in sequence). This dimensional change in the compression architecture allows the system to achieve high compression effectiveness while distributing complexity across multiple simpler components.
3Productivity
If grammar-based compressor with arithmetic coding is used, then compression effectiveness is improved, but ease of operation deteriorates
Solution Approach 1:
The patent segments the compression pipeline so that the complex grammar-based compressor with arithmetic coding is applied only to the output of the long-range compressor, not to the entire data stream. This segmentation limits the scope of complexity, making the system more manageable while still achieving high compression effectiveness on the residual data.
Solution Approach 2:
The patent applies the powerful but complex grammar-based compression with arithmetic coding only partially - specifically to the residual data after long-range compression. This partial application provides sufficient compression effectiveness for the remaining data without requiring the entire system to implement the full complexity of grammar-based compression.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A system (100) and method are provided for use with streaming blocks of data, each of the streaming blocks of data including a number bits of data. The system (100) includes a first compressor (106) and a second compressor (112). The first compressor (106) can receive and store a number n blocks of the streaming blocks of data, can receive and store a block of data to be compressed of the streaming blocks of data, can compress consecutive bits within the block of data to be compressed based on the n blocks of the streaming blocks of data, can output a match descriptor and a literal segment. The match descriptor is based on the compressed consecutive bits. The literal segment is based on a remainder of the number of bits of the data to be compressed not including the consecutive bits. The second compressor (112) can compress the literal segment and can output a compressed data block including the match descriptor and a compressed string of data based on the compressed literal segment.