Clustered Network Anomaly Detection for Lower Model Costs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training models for each element in a managed network or computerized system is computationally expensive and time-consuming, leading to outdated representations and inaccurate anomaly detection, especially when new elements are added, due to the high cost of data collection and storage.
Innovation Solution
Cluster similar elements together and train a single predictive model for the cluster, using hierarchical metadata or output records to reduce computational and storage costs, allowing new elements to be assigned to existing clusters without historical data, and apply local model execution to make immediate predictions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If individual predictive models are trained for each element in a managed network, then model accuracy for anomaly detection is improved, but computational cost and storage requirements increase significantly
Solution Approach 1:
The patent merges multiple individual element models into a single cluster-level predictive model. Elements with similar characteristics are grouped into clusters, and one model is trained per cluster rather than per element. This combining approach maintains detection accuracy while significantly reducing the number of models to train and store, thereby lowering computational cost and resource consumption.
Solution Approach 2:
The patent creates universal predictive models that serve multiple elements within a cluster simultaneously. A single model trained on aggregated data from multiple elements can predict anomalies for all elements in its cluster, making the model multi-functional. This universal approach eliminates the need for separate individual models while maintaining comprehensive anomaly detection coverage across diverse network elements.
2Measurement precision
If individual predictive models are trained for each element, then anomaly detection accuracy is improved, but the frequency of model retraining decreases due to computational constraints
Solution Approach 1:
By merging individual element models into cluster-level models, the patent reduces the total number of training operations required. This consolidation enables more frequent retraining cycles because the computational burden per retraining event is significantly lower. Models can be updated more often to reflect changing network conditions while maintaining accuracy.
Solution Approach 2:
The patent performs preliminary clustering of elements based on their characteristics before model training. This preliminary organization groups similar elements together, so that when retraining is needed, the model only needs to process aggregated data from the cluster rather than individual element data. This preparation enables faster, more frequent retraining while preserving detection accuracy.
3Measurement precision
If training data is collected for each new element before model training, then model accuracy is improved, but time to deploy models for new elements increases
Solution Approach 1:
The patent creates universal cluster models that can immediately serve new elements upon their addition to the network. When a new element is introduced, it is assigned to an existing cluster based on its characteristics, and the pre-trained cluster model is applied immediately. This eliminates the waiting period required to accumulate training data, enabling instant model deployment while maintaining accuracy through the model's ability to generalize across similar elements.
Solution Approach 2:
The patent performs preliminary model training on cluster-level aggregated data before new elements are added to the network. This advance preparation ensures that models are ready to deploy immediately when new elements arrive. The preliminary training on representative cluster data creates models that can generalize to new elements without requiring additional data collection time.
4Loss of time
If smaller amounts of training data are used for new elements, then model deployment time is reduced, but model accuracy decreases
Solution Approach 1:
The patent merges data from multiple elements within a cluster to create a comprehensive training dataset for each model. By aggregating data across similar elements rather than relying on data from a single element, the model receives sufficient training information even when individual element data is limited. This merging approach maintains accuracy while enabling faster deployment for new elements.
Solution Approach 2:
The patent trains universal models on aggregated cluster data that represents multiple element types and behaviors. This universal training approach creates models with broader generalization capability, allowing them to maintain accuracy when applied to new elements with limited individual data. The model learns patterns that are common across the cluster, making it robust to data quantity variations for individual elements.
Data Source
AI summary
Systems and methods are provided that include accessing a representation of a network that includes a plurality of elements; generating a plurality of clusters representative of the network, each cluster of the plurality of clusters including a respective non-overlapping subset of elements of the plurality of elements; obtaining, for each element of the subset of elements of a particular cluster of the plurality of clusters, historical data indicative of operation of at least two of the respective elements of the particular cluster; training, using the historical data, a model to detect anomalous activity in the particular cluster; obtaining operational data for a particular element of the subset of elements of the particular cluster; and determining, by applying the model to the operational data, that the particular element of the cluster exhibits anomalous activity.