DNN-Based Network Data Compression for Time-Series Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current compression techniques for network data, such as XOR-based compression and DeepSqueeze, fail to achieve sufficient compression ratios for time-series data, leading to high storage costs and inefficiencies due to their inability to effectively handle the large volumes of network telemetry data.
Innovation Solution
Deploying a Deep Neural Network (DNN) within the network environment to compress and reconstruct time-series data, allowing for significant reduction in storage requirements through predictive functionality and pruning techniques, enabling lossy compression with improved compression ratios.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If traditional compression techniques (XOR-based, GZIP, LZF, SZIP) are used on network time-series data, then lossless compression is achieved, but compression ratios are insufficient (only 1.18×-4.2×)
Solution Approach 1:
The patent changes the fundamental parameter of compression by transitioning from lossless to lossy compression methods. DeepSqueeze uses autoencoders to learn compressed representations that sacrifice some fidelity for dramatically higher compression ratios (10×-100×), while the DNN approach uses predictive modeling to reconstruct data with acceptable accuracy at much higher compression ratios than traditional methods
Solution Approach 2:
The patent replaces traditional mechanical compression algorithms (XOR operations, dictionary-based compression) with neural network-based approaches. DeepSqueeze substitutes autoencoder architectures for conventional compression algorithms, and the DNN approach substitutes predictive neural networks for traditional time-series compression methods, enabling the system to achieve superior compression ratios by learning complex patterns in the data
2Productivity
If DeepSqueeze autoencoder approach is used, then compression ratio improves (10×-100×), but computational overhead and training complexity increase significantly
Solution Approach 1:
The patent applies preliminary action by training the DNN model offline before deployment. The model is trained on historical network data to learn predictive patterns, and then this pre-trained model is deployed for compression. This separates the computationally intensive training phase from the operational compression phase, reducing real-time overhead while maintaining high compression ratios
Solution Approach 2:
The patent uses a simplified DNN architecture that is computationally efficient compared to full autoencoder systems. The approach uses lightweight predictive models that can be deployed with minimal computational resources, making the solution practical for network environments where resources may be constrained
3Measurement precision
If network data is stored for extended periods, then historical analysis fidelity is maintained, but storage costs become prohibitively expensive
Solution Approach 1:
The patent changes the storage parameter by compressing network data by orders of magnitude using DNN-based methods. Instead of storing raw high-volume data, the system stores compressed representations or model parameters that can be reconstructed when needed, reducing storage requirements from petabytes to manageable sizes while maintaining acceptable reconstruction accuracy for historical analysis
Solution Approach 2:
The patent extracts only the essential patterns and features from the network data using the trained DNN model. Rather than storing complete raw datasets, the system extracts and stores compressed representations that capture the essential information needed for historical analysis, significantly reducing storage requirements while preserving analytical value
Data Source
AI summary
Systems and methods for compressing network data are provided. According to one implementation, a method includes the step of collecting raw telemetry data from a network environment. The raw telemetry data is collected as time-series datasets. The method also includes the step of compressing the time-series datasets by deploying the time-series datasets as a Deep Neural Network (DNN) in the network environment itself. The time-series datasets are configured to be substantially reconstructed from the DNN using predictive functionality of the DNN.