Staged Data Compression with Block-Level Long-Range Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current long-range data compression techniques face scalability issues in systems with multiple communications terminals, requiring large compression and decompression caches, which leads to resource inefficiencies and increased memory needs.
Innovation Solution
A staged data compression approach involving block-level and byte-level compression stages, using hash tables to identify and remove duplicates, and a short-range compressor for residual compression, optimizing cache usage and processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If long-range data compression techniques are used to achieve high compression gain, then compression performance is improved, but memory requirements and device complexity increase significantly
Solution Approach 1:
The patent divides the compression system into two distinct components: a compression component that performs long-range compression with a large dictionary, and a decompression component that uses a much smaller cache. The large dictionary data is transmitted once and stored at the decompressor, while the compressor only needs to maintain a small buffer. This segmentation allows the compressor to achieve high compression ratios without requiring large memory at the compression end, resolving the contradiction between compression gain and memory requirements.
2Productivity
If large compression caches are used to exploit redundancies effectively, then compression performance is improved, but device complexity and resource requirements increase
Solution Approach 1:
The patent segments the dictionary/cache functionality between two devices: the compressor maintains only a small buffer for detecting redundancies, while the decompressor stores the large dictionary of historical data. This segmentation allows effective exploitation of redundancies through the large dictionary without increasing the complexity of the compressor device, which is typically the resource-constrained element.
Solution Approach 2:
The patent introduces an intermediary mechanism where the large dictionary is transmitted from the compressor to the decompressor once, and then the decompressor uses this dictionary for decompression without needing to maintain it at the compressor side. This intermediary approach allows the system to benefit from large dictionary coverage while keeping individual device complexity low.
3Productivity
If long-range compression with large dictionaries is implemented, then compression ratio is improved, but scalability to multiple terminals deteriorates
Solution Approach 1:
The patent segments the storage burden across multiple terminals: each terminal's decompressor maintains its own small cache, while the large dictionary data is distributed to terminals as needed. This segmentation enables the system to achieve high compression ratios for each terminal without requiring each terminal to have large memory resources, thereby improving scalability to multiple terminals.
Solution Approach 2:
The patent changes the operational parameters of the compression system by allowing the dictionary to be dynamically transmitted and cached at the decompressor side rather than being fixed at the compressor side. This parameter change enables flexible adaptation to different terminals and traffic patterns, improving scalability while maintaining high compression ratios.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Approaches for staged data compression are provided, where each stage reflects a progressive increase in granularity, resulting in a scalable approach that exhibits improved efficiency and compression performance. The first stage comprises a long-range block-level compressor that determines redundancies on a block-level basis (based on entire data blocks, as opposed to partial segments within data blocks). The second stage comprises a long-range byte-level compressor that compresses an uncompressed block based on byte segments within the block that match previously transmitted segments. The duplicate segments are replaced with pointers to matching segments within a decompressor cache. Nonmatching segments of the data block are left uncompressed and passed to a third stage short-range compressor (e.g., a grammar-based compressor). The staged progression in granularity provides advantages of maximizing the compression gain while minimizing processing and storage requirements of the compressor and decompressor.