Compressed Telemetry Transmission for Network ML Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning systems in network assurance require large amounts of time series data to be transmitted from networking devices to the cloud, overwhelming network resources due to high frequency reporting, which is not efficiently managed by existing compression methods.
Innovation Solution
A service jointly trains a compression model and an inference model to compress telemetry data, deploying the compression model to the network to reduce data transmission load while maintaining predictive accuracy for machine learning-based operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If large amounts of time series data are transmitted from networking devices to the cloud for machine learning analysis, then predictive accuracy and anomaly detection capability are improved, but network resources are overwhelmed and data transmission efficiency deteriorates
Solution Approach 1:
The patent extracts and transmits only the most relevant and informative features from the raw time series data after compression, rather than transmitting all raw data. The compression model identifies and retains key patterns while discarding redundant information, achieving a balance between predictive accuracy and network resource consumption.
Solution Approach 2:
The patent changes the parameter representation of the data by transforming raw time series data into compressed feature representations. The compression model learns optimal parameter transformations that preserve predictive information while reducing data dimensionality and transmission requirements.
2Loss of energy
If data compression is applied to reduce transmission load, then network resource usage is optimized, but data transmission fidelity and machine learning model performance may deteriorate
Solution Approach 1:
The compression model is dynamically trained and adjusted based on the specific characteristics of the time series data and the requirements of the downstream machine learning models. The compression level and method can be adapted to maintain optimal fidelity for different data patterns and prediction tasks.
Solution Approach 2:
The system employs feedback mechanisms where the performance of the compressed data in downstream machine learning models is evaluated, and this information is used to refine and retrain the compression model. This closed-loop approach ensures that compression fidelity is continuously optimized to meet predictive accuracy requirements.
3Reliability
If high frequency data reporting is implemented to capture detailed network patterns, then anomaly detection capability is improved, but the volume of telemetry data increases and overwhelms the network
Solution Approach 1:
The compression model extracts only the essential patterns and features from high-frequency data that are necessary for anomaly detection, filtering out redundant information. This allows the system to maintain high anomaly detection capability while transmitting a reduced volume of compressed data.
Solution Approach 2:
The system applies partial compression to the most critical data streams while maintaining higher fidelity for particularly important metrics. The compression level is adjusted based on the relative importance of different data sources for anomaly detection, applying more aggressive compression to less critical data.
Data Source
AI summary
In one embodiment, a service receives telemetry data indicative of a plurality of performance metrics captured in a network. The service jointly trains, using the received telemetry data, a compression model and an inference model, the compression model being a first machine learning model trained to convert the telemetry data into a compressed representation of the telemetry data and the inference model being a second machine learning model trained to take the compressed representation of the telemetry data as input and apply a classification label to it. The service deploys the compression model to the network. The service receives compressed telemetry data generated by the compression model deployed to the network. The service uses the inference model to classify the compressed telemetry data generated by the compression model deployed to the network.