Shared Compression Histories for Long-Match Network Traffic
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression methods face inefficiencies due to insufficiently long matches between input streams and compression histories, leading to poor compression ratios and increased processing overhead, especially in devices with long access latencies and unsynchronized compression histories across multiple devices.
Innovation Solution
The system improves compression efficiency by using algorithms to identify long matches in compression histories, synchronizing storage, and sharing compression histories and indexes across devices to leverage previously transmitted data for network traffic compression.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of stationary object
If compression histories are stored on disk with long access latencies, then storage capacity is improved, but access speed deteriorates
Solution Approach 1:
The compression history is segmented into two parts: frequently accessed recent data stored in fast memory (RAM) and older data stored on disk. This segmentation allows the system to maintain large storage capacity while providing fast access for active compression operations.
Solution Approach 2:
A memory buffer acts as an intermediary between the disk storage and the compression algorithm. The buffer caches recently accessed compression history data, mediating between the slow disk and the fast compression process to reduce access latency.
2Productivity
If multiple processes access compression history simultaneously, then throughput is improved, but processing overhead increases
Solution Approach 1:
The compression history is divided into multiple segments or partitions that can be accessed by different processes simultaneously. This segmentation reduces contention and allows parallel access without requiring complex locking mechanisms for the entire history.
Solution Approach 2:
Multiple compression histories from different device pairs are merged into a shared cache structure, allowing processes to benefit from compression data of other connections while reducing overall system overhead through consolidated management.
3Productivity
If compression histories are synchronized across devices, then compression ratio is improved, but network overhead increases
Solution Approach 1:
Instead of synchronizing entire compression histories, devices copy only the essential index information and metadata about the compression history. The actual compression data remains local, reducing network overhead while maintaining sufficient coordination for effective compression.
Solution Approach 2:
Compression histories are pre-synchronized or pre-coordinated during idle periods or during connection establishment, rather than continuously during active data transfer. This preliminary action reduces the need for frequent synchronization overhead during high-bandwidth operations.
Data Source
AI summary
Systems and methods of storing previously transmitted data and using it to reduce bandwidth usage and accelerate future communications are described. By using algorithms to identify long compression history matches, a network device may improve compression efficiently and speed. A network device may also use application specific parsing to improve the length and number of compression history matches. Further, by sharing compression histories and compression history indexes across multiple devices, devices can utilize data previously transmitted to other devices to compress network traffic. Any combination of the systems and methods may be used to efficiently find long matches to stored data, synchronize the storage of previously sent data, and share previously sent data among one or more other devices.