Compression History Sharing for Long-Match Network Traffic

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current data compression methods face inefficiencies due to short matches between input streams and compression histories, leading to poor compression ratios and increased processing overhead, especially in multi-stream transmissions and devices with storage-based compression histories. Additionally, synchronization of compression histories across devices is challenging, and existing methods do not effectively leverage previously transmitted data for compression between different device clusters.

Innovation Solution

The system improves compression efficiency by using algorithms to identify long matches in compression histories, synchronizing and sharing compression histories across devices, and employing application-specific parsing to enhance match lengths. It determines disk-based compression based on match thresholds, prioritizes matching fingerprints, removes application layer protocol headers, and establishes an ad-hoc hierarchy of caches to optimize data transmission.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If compression histories are stored on disk-based media, then storage capacity is increased, but access latency increases significantly

Engineering Contradiction:

Improvestorage capacityVSAvoidaccess latency

Core Design Contradiction:

Quantity of substanceVSLoss of time

Solution Approach 1:

The compression history is segmented into two parts: frequently accessed recent data is stored in memory for fast access, while older data is stored on disk for long-term retention. This segmentation allows the system to maintain large storage capacity while minimizing access latency for active compression operations.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

A memory buffer acts as an intermediary between the disk-based compression history storage and the compression algorithm. The memory buffer pre-loads and caches compression history data, mediating between the slow disk storage and the fast compression processing, thereby reducing access latency.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If multiple data streams are transmitted simultaneously, then network throughput is improved, but compression history access overhead increases

Engineering Contradiction:

Improvenetwork throughputVSAvoidcompression history access overhead

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The system maintains separate compression history instances for different data streams, segmenting the compression state per-stream. This allows parallel compression operations on multiple streams without contention, improving network throughput while managing access overhead through organized separation.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system selectively applies compression to only those portions of multiple data streams that have matching history, rather than attempting to compress entire streams uniformly. This partial action approach reduces processing overhead while maintaining high throughput for compressible data.

Inventive Principle:

Principle #16Partial or excessive action

3Productivity

If compression history is synchronized between devices, then compression efficiency is improved, but synchronization overhead increases

Engineering Contradiction:

Improvecompression efficiencyVSAvoidsynchronization overhead

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The system extracts and transmits only the essential synchronization information (such as history tokens or references) rather than synchronizing entire compression histories between devices. This extraction approach maintains compression efficiency while minimizing synchronization overhead.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

Instead of maintaining identical compression histories on multiple devices, the system uses copying mechanisms where reference copies or tokens are exchanged between devices. This allows compression efficiency through history sharing while reducing overhead compared to full synchronization.

Inventive Principle:

Principle #26Copying

4Speed

If short matches are used in compression, then processing speed is improved, but compression ratio deteriorates

Engineering Contradiction:

Improveprocessing speedVSAvoidcompression ratio

Core Design Contradiction:

SpeedVSLoss of substance

Solution Approach 1:

The system performs multiple passes or uses progressive matching where shorter initial matches are found quickly for immediate compression, then longer matches are searched for in subsequent passes. This partial action approach balances processing speed with compression ratio by applying different match lengths selectively.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS7532134B2Systems and methods for sharing compression histories between multiple devices

Publication Date: 2009.05.12 CITRIX SYSTEMS INC

AI summary

Systems and methods of storing previously transmitted data and using it to reduce bandwidth usage and accelerate future communications are described. By using algorithms to identify long compression history matches, a network device may improve compression efficiently and speed. A network device may also use application specific parsing to improve the length and number of compression history matches. Further, by sharing compression histories and compression history indexes across multiple devices, devices can utilize data previously transmitted to other devices to compress network traffic. Any combination of the systems and methods may be used to efficiently find long matches to stored data, synchronize the storage of previously sent data, and share previously sent data among one or more other devices.