Unsupervised Anomaly Detection via t-digest Tolerance Intervals
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly detection systems in cloud and enterprise environments face challenges in efficiently identifying anomalies due to varying behavioral patterns across different systems, leading to false positives and neglect of actual anomalies, especially with large-scale systems exhibiting trends and seasonal fluctuations, which complicates the selection of optimal thresholds.
Innovation Solution
The system employs a streaming capability for quantile estimates using a t-digest structure and non-parametric tolerance intervals, allowing for real-time updates and forgetting of older data points, enabling unsupervised anomaly detection without requiring storage of all data points, thus reducing resource intensity and improving scalability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If threshold-based alerting is used with user-defined thresholds, then domain knowledge can be injected to supervise anomaly detection, but the system produces false positive alerts and cannot keep up with evolving normal behavioral patterns
Solution Approach 1:
The system performs self-service by automatically learning normal behavioral patterns from historical data and dynamically adjusting thresholds without requiring continuous user intervention. The anomaly detection model trains itself on incoming data streams, adapting to evolving patterns while users only need to configure initial parameters.
Solution Approach 2:
The system dynamically changes detection parameters (thresholds, sensitivity levels) based on learned behavioral patterns. Instead of fixed user-defined thresholds, the system adapts parameters in response to detected anomalies and changing system conditions, improving reliability while reducing the complexity of manual parameter management.
2Measurement precision
If all data points are stored for anomaly detection, then complete historical context is available for analysis, but resource intensity increases and scalability is reduced
Solution Approach 1:
The system extracts only the essential features and patterns from historical data that are necessary for anomaly detection, rather than storing and processing all raw data points. By identifying and retaining only the most informative characteristics, the system maintains detection precision while significantly reducing resource consumption and improving scalability.
3Ease of manufacture
If static tolerance intervals are used for anomaly detection, then implementation is simple, but the system cannot adapt to trends and seasonal fluctuations in system behavior
Solution Approach 1:
The system transitions from static to dynamic tolerance intervals that automatically adjust to captured behavioral patterns including trends and seasonal variations. The model continuously learns from incoming data and updates detection boundaries in real-time, maintaining ease of implementation through automated adaptation while significantly improving versatility in handling diverse behavioral patterns.
Data Source
AI summary
Systems and methods for unsupervised training and evaluation of anomaly detection models are described. In some embodiments, an unsupervised process comprises generating an approximation of a data distribution for a training dataset including varying values for a metric of a computing resource. The process further determines, based on the size of the training dataset, a first quantile probability and a second quantile probability that represent an interval for covering a prescribed proportion of values for the metric within a prescribed confidence level. The process further trains a lower limit of the anomaly detection model using a first quantile that represents the first quantile probability in the approximation of the data distribution and an upper limit using a second quantile that represents the second quantile probability in the approximation. The trained upper and lower limits may be used to monitor input data for anomalous behavior and, if detected, trigger responsive action(s).