Shared Compression Histories for Long-Match Network Data Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data compression methods face inefficiencies due to insufficiently long matches between input streams and compression histories, leading to poor compression ratios and increased processing overhead, especially in multi-stream transmissions and when compression histories are not synchronized across devices.
Innovation Solution
The system employs algorithms to identify long matches in compression histories by creating indexes in memory, synchronizing compression histories across devices, and sharing compression indexes to improve compression efficiency and reduce bandwidth usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If compression histories are stored on disk with long access latencies, then storage capacity is improved, but compression speed and processing efficiency deteriorate due to multiple disk accesses required
Solution Approach 1:
The patent segments compression histories into multiple partitions stored on disk, while maintaining active compression history segments in memory. This segmentation allows the system to store large volumes of compression data (improving storage capacity) while keeping frequently accessed segments in fast memory (maintaining compression speed). The system selectively accesses only relevant partitions based on match length thresholds, reducing the number of disk accesses required.
2Productivity
If multiple data streams are transmitted simultaneously, then network throughput is improved, but compression efficiency deteriorates due to increased processing overhead and multiple compression history accesses
Solution Approach 1:
The patent establishes indexes for multiple compression history partitions in advance, before compression operations begin. These indexes pre-organize the location information of compression data across partitions. When multiple data streams require simultaneous compression, the system can quickly locate matches by consulting pre-built indexes rather than searching through uncompressed history data, significantly reducing processing overhead while maintaining high network throughput.
3Reliability
If compression histories are synchronized between devices, then compression reliability is improved, but network overhead increases due to transmission of synchronization information
Solution Approach 1:
The patent extracts and transmits only the essential synchronization information needed for compression history alignment between devices. Rather than synchronizing entire compression histories, the system identifies and transmits only the critical location identifiers and match length information from indexes. This selective extraction maintains compression reliability while minimizing network overhead, as only the necessary metadata for synchronization is transmitted.
4Adaptability or versatility
If short matches are used in compression, then compression flexibility is improved, but compression ratio deteriorates and processing overhead increases
Solution Approach 1:
The patent implements a dynamic compression strategy that adapts match length selection based on data characteristics and performance requirements. The system can dynamically adjust between short matches (providing flexibility for diverse data patterns) and long matches (providing high compression ratios). The dynamic index structure allows the system to efficiently search for matches of varying lengths without sacrificing compression ratio or processing efficiency, as the index organizes data to enable quick access regardless of match length.
Data Source
AI summary
Systems and methods of storing previously transmitted data and using it to reduce bandwidth usage and accelerate future communications are described. By using algorithms to identify long compression history matches, a network device may improve compression efficiently and speed. A network device may also use application specific parsing to improve the length and number of compression history matches. Further, by sharing compression histories, compression history indexes and caches across multiple devices, devices can utilize data previously transmitted to other devices to compress network traffic. Any combination of the systems and methods may be used to efficiently find long matches to stored data, synchronize the storage of previously sent data, and share previously sent data among one or more other devices.