Two-Stage Data Compression for Long- and Short-Range Redundancy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current lossless data compression techniques, such as Lempel-Ziv family compressors, are inefficient in exploiting long-range redundancies in internet traffic and cannot compress already compressed entities, leading to suboptimal performance and increased data size.
Innovation Solution
A two-stage compression system comprising a long range compressor front end that captures macro redundancies using a large history buffer and a short range compressor back end that employs grammar transforms and adaptive arithmetic coding to exploit micro redundancies, allowing for efficient compression of entities that have already been compressed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If Lempel-Ziv family compressors are used with small dictionary size, then device complexity is reduced, but compression efficiency deteriorates due to inability to exploit long-range redundancies
Solution Approach 1:
The patent divides the compression task into two distinct stages: a long-range compressor that handles macro redundancies using a large history buffer, and a short-range compressor that handles micro redundancies using grammar transforms. This segmentation allows each stage to be optimized independently, resolving the contradiction between complexity and efficiency.
Solution Approach 2:
The patent introduces an intermediary component that transfers data between the long-range and short-range compressors. This intermediary enables the combination of both compression approaches, allowing the system to achieve high compression efficiency while managing complexity through modular architecture.
2Productivity
If history window size is increased to capture long-range redundancies, then compression efficiency improves, but device complexity increases making the system impractical
Solution Approach 1:
The patent segments the compression function into two parts: long-range compression handling macro redundancies and short-range compression handling micro redundancies. This segmentation allows the system to achieve high compression efficiency without requiring a single overly complex compressor, thus resolving the contradiction.
Solution Approach 2:
The patent transitions from a single-dimension approach (one compressor with large history window) to a two-dimension approach (two-stage compression system). This dimensional change allows the system to exploit both long-range and short-range redundancies effectively while managing complexity through architectural design.
3Productivity
If compression techniques are applied to already compressed entities, then compression efficiency may improve, but data size increases due to inability to compress compressed data
Solution Approach 1:
The patent changes the approach to handling already-compressed data by using grammar-based compression in the second stage. This parameter change in compression methodology allows the system to achieve compression efficiency on compressed entities without increasing data size, as the grammar-based approach is specifically designed to handle such cases.
Data Source
AI summary
A system and method are provided for use with streaming blocks of data, each of the streaming blocks of data including a number bits of data. The system includes a first compressor and a second compressor. The first compressor can receive and store a number n blocks of the streaming blocks of data, can receive and store a block of data to be compressed of the streaming blocks of data, can compress consecutive bits within the block of data to be compressed based on the n blocks of the streaming blocks of data, can output a match descriptor and a literal segment. The match descriptor is based on the compressed consecutive bits. The literal segment is based on a remainder of the number of bits of the data to be compressed not including the consecutive bits. The second compressor can compress the literal segment and can output a compressed data block including the match descriptor and a compressed string of data based on the compressed literal segment.