Clustered Network Anomaly Detection for Low-Cost Model Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training models for each element in a managed network or computerized system is computationally expensive and resource-intensive, leading to outdated representations and inaccurate anomaly detection, especially when new elements are added, due to the need for significant training data and computational resources.
Innovation Solution
Cluster similar elements together and train a single predictive model for the cluster, using hierarchical metadata or output records to reduce computational and storage costs, allowing new elements to be assigned to existing clusters without historical data, and apply local model execution to improve accuracy and frequency of updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If individual predictive models are trained for each element in a managed network, then model accuracy for anomaly detection is improved, but computational cost and resource consumption increase significantly
Solution Approach 1:
The patent combines multiple individual element models into a single cluster-level predictive model. By aggregating data from multiple elements and training one model at the cluster level, the system maintains anomaly detection accuracy while significantly reducing computational cost and resource consumption compared to training separate models for each element.
Solution Approach 2:
The cluster-level predictive model serves multiple elements simultaneously, making it a universal model that can detect anomalies across different elements within the same cluster. This multi-functional approach allows a single model to replace multiple individual models, reducing overall computational burden while maintaining detection capability across diverse elements.
2Measurement precision
If individual predictive models are trained for each element, then element-specific anomaly detection accuracy is improved, but the frequency of model retraining decreases due to resource constraints
Solution Approach 1:
By merging individual element models into a single cluster-level model, the system reduces the total number of training operations required. This consolidation enables more frequent retraining cycles at the cluster level, ensuring models stay current with changing patterns while consuming fewer computational resources than frequent individual model retraining would require.
3Loss of time
If smaller amounts of training data are used for new elements, then model deployment time is reduced, but model accuracy decreases
Solution Approach 1:
The cluster-level predictive model provides a pre-trained universal model that can be immediately applied to new elements upon clustering, eliminating the need to accumulate training data for each new element. This approach enables immediate deployment with acceptable accuracy, as the model learns patterns from aggregated cluster data rather than requiring element-specific training datasets.
4Ease of manufacture
If clustering algorithms are applied to past outputs to group elements, then elements can be analyzed in common, but significant memory and computational resources are required
Solution Approach 1:
The patent extracts essential clustering information from element metadata and characteristics without requiring full application of computationally intensive clustering algorithms to historical output data. By taking out only the necessary grouping information from metadata, the system achieves common analysis capability while avoiding the significant memory and computational resource requirements of traditional clustering methods.
Data Source
AI summary
Systems and methods are provided that include accessing a representation of a network that includes a plurality of elements; generating a plurality of clusters representative of the network, each cluster of the plurality of clusters including a respective non-overlapping subset of elements of the plurality of elements; obtaining, for each element of the subset of elements of a particular cluster of the plurality of clusters, historical data indicative of operation of at least two of the respective elements of the particular cluster; training, using the historical data, a model to detect anomalous activity in the particular cluster; obtaining operational data for a particular element of the subset of elements of the particular cluster; and determining, by applying the model to the operational data, that the particular element of the cluster exhibits anomalous activity.