Shared Compression Histories for Long-Match Network Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression methods face inefficiencies due to insufficiently long matches between input streams and compression histories, leading to poor compression ratios and increased processing overhead, especially in multi-data stream transmissions and devices with storage-based compression histories. Additionally, synchronization of compression histories across devices is challenging, and benefits are limited to repeated data transmissions between the same devices without explicit hierarchies.
Innovation Solution
The system employs algorithms to identify long compression history matches by creating indexes in memory, synchronizes compression histories by prioritizing recently accessed data, and shares compression histories across devices to leverage previously transmitted data for efficient compression, even across different device clusters without requiring explicit hierarchies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If compression histories are stored on disk-based storage media, then storage capacity is increased, but access latency increases and compression efficiency decreases
Solution Approach 1:
The compression history is segmented into two storage locations: recently accessed data stored in memory for fast access, and older data stored on disk for capacity. This segmentation allows the system to benefit from both fast memory access for frequent operations and large disk capacity for historical data retention.
Solution Approach 2:
Memory serves as an intermediary between the compression algorithm and disk-based storage. The memory cache holds frequently accessed compression history data, mediating between the fast but limited memory and the slow but large-capacity disk storage, thereby reducing overall access latency.
2Adaptability or versatility
If multiple processes access compression history simultaneously, then multi-data stream transmission is enabled, but processing overhead and access conflicts increase
Solution Approach 1:
The compression history data structure is segmented with dedicated fields (such as length fields and offset fields) that allow multiple processes to independently parse and access different portions of the data simultaneously without requiring complex locking mechanisms, thereby reducing processing overhead.
Solution Approach 2:
Each process receives a copy of the relevant compression history data or metadata in memory, allowing simultaneous access without contention. This copying approach enables multi-stream transmission while avoiding the complexity of synchronized access to shared resources.
3Productivity
If compression histories are synchronized across devices, then compression benefits are shared, but synchronization complexity and storage overhead increase
Solution Approach 1:
The patent extracts only the essential synchronization information (such as data length, offset, and fingerprint identifiers) from the full compression history and transmits this metadata between devices. This extraction approach enables compression benefits to be shared across devices without the complexity of synchronizing entire compression histories.
Solution Approach 2:
A standardized metadata format acts as an intermediary representation of compression history data that can be efficiently exchanged between devices. This intermediary format captures the essential information needed for compression without requiring full data synchronization, reducing both complexity and storage overhead.
4Speed
If short matches are used in compression, then processing speed is maintained, but compression ratios deteriorate
Solution Approach 1:
The system performs preliminary actions by pre-computing and storing metadata about compression history data (such as length, offset, and fingerprint information) before actual compression occurs. This preliminary organization enables the system to quickly identify long matches during compression without sacrificing processing speed, thereby achieving both good compression ratios and maintained throughput.
Data Source
AI summary
Systems and methods of storing previously transmitted data and using it to reduce bandwidth usage and accelerate future communications are described. By using algorithms to identify long compression history matches, a network device may improve compression efficiently and speed. A network device may also use application specific parsing to improve the length and number of compression history matches. Further, by sharing compression histories and compression history indexes across multiple devices, devices can utilize data previously transmitted to other devices to compress network traffic. Any combination of the systems and methods may be used to efficiently find long matches to stored data, synchronize the storage of previously sent data, and share previously sent data among one or more other devices.