Multi-contextual Anomaly Detection via Model Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly detection systems often fail to provide early warnings effectively, leading to late issue identification and false positive or false negative results due to the massive scale of data analysis, statistical fluctuations, and overfitting models, which can cause undue burden on systems and result in network downtimes or missed anomalies.
Innovation Solution
A system that aggregates outputs from multiple machine learning models trained on different behavioral patterns of a metric, generating an aggregate anomaly score based on seasonality, rare occurrence, and combined metric behaviors, along with a duration score to distinguish real anomalies from statistical fluctuations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a single machine learning model is used for anomaly detection, then the system complexity is low, but the detection accuracy is reduced due to false positives and negatives
Solution Approach 1:
The patent divides the anomaly detection task into multiple specialized machine learning models, each trained to detect specific types of anomalies or patterns. Instead of using one general model, the system segments the detection function across multiple models that can be selectively applied based on the data characteristics, thereby improving detection accuracy while managing complexity through modular design
Solution Approach 2:
The patent creates a universal anomaly detection system that can handle multiple types of anomalies through a single framework. The system uses a common infrastructure with multiple machine learning models that can be configured for different detection scenarios, making the system adaptable to various anomaly types without requiring separate dedicated systems for each case
2Loss of time
If the detection system is tuned to be more sensitive, then early anomaly detection is improved, but false positive warnings increase
Solution Approach 1:
The patent implements dynamic thresholding and model selection mechanisms that adjust the sensitivity of anomaly detection based on current data conditions. The system can dynamically select which machine learning models to apply and adjust detection thresholds in real-time, allowing early detection when appropriate while reducing false positives when data patterns suggest normal variations
Solution Approach 2:
The system incorporates feedback mechanisms where detection results and system performance are continuously monitored. This feedback is used to refine model parameters, adjust sensitivity thresholds, and improve the distinction between true anomalies and false positives over time, thereby maintaining early detection capability while reducing false alarm rates
3Reliability
If the detection system is tuned to be more restrictive, then false positive warnings are reduced, but false negative reporting increases
Solution Approach 1:
By segmenting the detection function across multiple specialized models, the system can apply different sensitivity levels to different anomaly types. Restrictive tuning for one model does not necessarily affect others, allowing the system to maintain high reliability for common anomalies while preserving detection capability for rare but critical events through appropriate model selection
4Measurement precision
If multiple machine learning models are used for anomaly detection, then detection accuracy is improved, but the computational burden increases
Solution Approach 1:
The patent segments the computational workload across multiple specialized models rather than using one comprehensive model. This segmentation allows for more efficient computation by applying only the necessary models to specific data types, reducing overall computational burden while maintaining high detection accuracy through targeted analysis
Solution Approach 2:
The system dynamically changes parameters such as model selection, data sampling rates, and detection thresholds based on current computational conditions and data characteristics. This allows the system to adjust its computational intensity to match available resources while maintaining effective anomaly detection through adaptive parameter optimization
Data Source
AI summary
The disclosure relates to systems and methods of detecting anomalies using a plurality of machine learning models. Each of the machine learning models may be trained to detect a respective behavior of historical data values for a given metric. Thus, a system may perform anomaly detection based on different behaviors of the same metric of data, reducing instances of false positive anomaly detection while also reducing instances of false negative reporting. The plurality of machine learning models may be trained to detect anomalies across multiple different types of metrics as well, providing robust multi-metric anomaly detection across a range of behaviors of historical data values. The system may implement a pluggable architecture for the plurality of machine learning models in which models may be added or removed from pluggable architecture. In this way, the system may detect anomalies using a configurable set of machine learning models.