Network Traffic Table Compression Using Offline Grouping Plans
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression methods for network traffic data fail to achieve high compression ratios and fast compression times, especially in high-speed links, due to their inability to effectively utilize structural similarities within the data, leading to inefficient storage and transmission.
Innovation Solution
The proposed group compression method formats network data as a table and generates a compression plan based on structural similarities learned during a training period, allowing for the grouping of columns to minimize compressed size, which can be applied on an ongoing basis, achieving better compression ratios than general compressors like GZIP.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If general compression methods like GZIP are used on network traffic data, then compression is simple to implement, but the compression ratio is insufficient and storage space is not adequately reduced
Solution Approach 1:
The patent segments network traffic data into discrete records with fixed columns, treating each column as an independent compressible unit. This segmentation allows the compression algorithm to identify and exploit structural similarities across records more effectively than treating the data as a continuous stream, thereby achieving better compression ratios without excessive complexity.
Solution Approach 2:
The patent performs preliminary analysis during a training period to learn the structural characteristics of network traffic data before actual compression. This preliminary action includes identifying frequently occurring values and patterns in each column, which are then used to optimize the compression strategy for subsequent data, improving compression efficiency without adding complexity to the main compression path.
2Quantity of substance
If compression algorithms analyze all data before compression, then compression ratio can be optimized, but compression time increases and real-time monitoring is compromised
Solution Approach 1:
The patent performs compression optimization in advance during a training period by analyzing a subset of data to learn structural patterns. This preliminary analysis identifies the most effective compression strategies for each column based on observed data characteristics. Once learned, these strategies are applied to subsequent data without requiring re-analysis, thus achieving high compression ratios with minimal real-time processing delay.
Solution Approach 2:
The patent analyzes only a subset of data during the training period rather than all data, which is sufficient to learn the structural patterns needed for effective compression. This partial analysis approach achieves the necessary optimization without the time cost of analyzing the entire dataset, enabling real-time monitoring capabilities to be maintained.
3Quantity of substance
If flow-based TCP/IP header compression is used to reduce header size, then transmission bandwidth is improved, but the structural information within records is not utilized
Solution Approach 1:
The patent segments network records into distinct columns, allowing independent compression of each column based on its specific characteristics. This segmentation enables the exploitation of structural similarities within each column across multiple records, going beyond traditional flow-based compression that only handles header information. The result is more effective utilization of structural information while maintaining improved transmission bandwidth.
4Quantity of substance
If table compression is used to compress network data, then storage space is reduced, but the algorithm requires scanning all rows before compression which increases latency
Solution Approach 1:
The patent performs the computationally intensive analysis of data patterns and structural characteristics during a training period before actual compression begins. This preliminary action pre-determines the optimal compression strategy for each column based on observed data. During real-time operation, only the pre-determined compression rules need to be applied, dramatically reducing latency while maintaining effective storage space reduction.
Data Source
AI summary
An important component of network monitoring is to collect traffic data which is a bottleneck due to large data size. We introduce a new table compression method called “Group Compression” to address this problem. This method uses a small training set to learn the relationship among columns and group them; the result is a “compression plan”. Based on this plan, each group is compressed separately. This method can reduce the compressed size to 60%-70% of the IP flow logs compressed by GZIP.