Packet-Level Clustering for Memory-Assisted Network Traffic Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data compression systems are inefficient in capturing redundancy in network data packets due to their inability to account for statistical redundancies within packets and dependencies across packets, leading to sub-optimal compression performance.
Innovation Solution
A memory-assisted data compression system that uses packet-level clustering, where data packets are vectorized, clustered based on calculated distances, and compressed using algorithms like Lite PAQ or CTW, with indices representing the training data packet cluster included for efficient transmission.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional compression techniques are used on individual packets, then compression can be performed on each packet independently, but compression performance is poor because sequence length is too short to capture redundancy effectively
Solution Approach 1:
The patent merges multiple individual packets into a combined sequence for compression processing. By treating a group of packets as a single compression unit, the system captures cross-packet redundancies that would be missed when compressing packets individually, thereby improving compression performance without requiring fundamentally new compression algorithms
Solution Approach 2:
The patent performs preliminary buffering and sorting of packets before compression. Packets are collected, sorted by destination address, and organized into sequences in advance of the actual compression operation. This preliminary organization enables the compression algorithm to work on optimally structured data, capturing redundancies more effectively
2Loss of substance
If memory-assisted deduplication is used to eliminate redundancy, then some packet-level redundancy can be removed, but statistical redundancies within packets and dependencies across packets are not accounted for
Solution Approach 1:
The patent changes the operational parameters of compression by adjusting the sequence length to span multiple packets and by modifying the sorting criteria to group packets by destination address. These parameter changes enable the compression algorithm to capture both within-packet and across-packet redundancies, significantly improving compression efficiency beyond what memory-assisted deduplication alone can achieve
3Productivity
If packets are compressed individually without clustering, then processing is simpler and faster, but compression rates are sub-optimal due to inability to exploit cross-packet dependencies
Solution Approach 1:
The patent segments the network traffic stream into distinct groups of packets that share common characteristics, particularly destination address. By dividing the traffic into these segments and compressing each segment separately after sorting, the system exploits cross-packet dependencies within each segment while maintaining manageable processing complexity through the structured segmentation approach
Data Source
AI summary
Certain implementations of the disclosed technology may include methods and computing systems for memory-assisted compression of network packets using packet-level clustering. According to an example implementation, a method is provided. The method may include vectorizing a plurality of data packets stored in a memory, calculating respective distances between each of the respective vectorized data packets, clustering the plurality of data packets into a plurality of data packet clusters, obtaining a sample data packet to be compressed, identifying a training data packet cluster from among the plurality of data packet clusters, and compressing the sample data packet using a compression algorithm.