Dynamic Metric Thresholds for Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional anomaly detection methods face challenges in accurately defining baseline thresholds, leading to false alerts and missed genuine anomalies due to the inability to capture temporal behavior and spatial interactions, especially in complex system environments.
Innovation Solution
A method and system for generating and optimizing metric thresholds that derive temporal and spatio-temporal properties of metrics, using static, dynamic, and composite thresholds, and employing self-learning and self-tuning mechanisms to adapt to changes and user feedback, ensuring accurate anomaly detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If blanket threshold is set by using knowledge of experts or simple statistical measures, then implementation is simple, but it results in alarm deluge and false positives
Solution Approach 1:
The patent implements dynamic thresholds that adapt to temporal behavior patterns of metrics. Instead of static blanket thresholds, the system learns temporal properties (trend, seasonality, periodicity) and adjusts thresholds dynamically based on predicted normal behavior, reducing false positives while maintaining detection accuracy
Solution Approach 2:
The system performs self-learning by automatically deriving temporal properties from historical data without requiring expert knowledge. The anomaly detection model trains itself on available metrics to establish baseline behavior patterns, eliminating the need for manual threshold configuration while improving reliability
2Ease of manufacture
If existing unsupervised methods such as K-means clustering and density estimation are used, then implementation is straightforward, but they cannot capture temporal behavior across different time stamps
Solution Approach 1:
The patent employs dynamic thresholding that evolves with temporal patterns. The system decomposes time series into trend, seasonality, and residual components, then establishes thresholds based on the residual behavior after removing predictable patterns, enabling accurate detection of temporal anomalies
Solution Approach 2:
The system transforms the problem from static clustering to temporal dimensionality analysis by decomposing time series into multiple components (trend, seasonality, residuals) and analyzing anomalies in the residual dimension where temporal patterns are removed, capturing temporal behavior that traditional methods miss
3Measurement precision
If ARIMA and LSTM models are used to capture temporal behavior, then temporal patterns are captured, but they are susceptible to data noise and excessive noise leads to increase in false positives
Solution Approach 1:
The patent segments the time series signal into distinct components: trend, seasonality, and residuals. By separating the predictable temporal patterns (trend and seasonality) from the unpredictable variations (residuals), the system establishes thresholds only on the residual component, effectively filtering out noise from regular temporal patterns and reducing false positives
Solution Approach 2:
The system extracts and removes predictable temporal components (trend and seasonality) from the raw time series data before performing anomaly detection. This extraction isolates the genuine anomalies in the residual signal, making the detection more robust to noise in the original data
4Ease of operation
If anomaly detection is applied independently to multiple time series, then each series is analyzed individually, but correlation across time series is ignored
Solution Approach 1:
The patent merges multiple time series analyses by establishing a global threshold model that learns from all available metrics simultaneously. The system captures correlations across time series through joint training, allowing anomalies in one metric to inform the detection in related metrics, preserving correlation information while maintaining operational simplicity
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The present disclosure is related to the field of threshold generation where complex temporal and spatial behavior of metrics are modelled to define normal operating baseline of every component in a system environment. Embodiments of the present disclosure provide methods and systems for generation and optimization of metric threshold for anomaly detection. In the present disclosure, temporal properties such as variation, behavioral patterns of a plurality of metrics are derived. The derived temporal properties are used to generate data-driven and domain-aware static or dynamic thresholds. Additionally, the present disclosure factors spatial and temporal properties collectively to mine a role of influencing metrics and define composite thresholds. In order to cater to dynamic behavior of the system environment and changes in business and technological aspects, the system and method of the present disclosure self-learns, self-tunes, and adapts itself based on user feedback which helps in capturing tacit knowledge of domain experts.