Anomaly Detection via Dynamic Thresholds on Time-Series Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly detection techniques in 5G networks face challenges in accurately identifying outliers in multivariate time-series datasets without relying on labeled data, as they often assume normal distributions and fail to adapt to dynamic patterns, leading to high false positives and negatives.
Innovation Solution
A method using historic error values to determine if a data element is an outlier by predicting its value based on previous data elements, calculating a threshold value from these errors, and comparing it to the actual value, allowing for dynamic threshold adjustment and multivariate analysis without requiring labeled data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If parametric methods are used to determine threshold values based on distribution assumptions, then the threshold determination becomes systematic, but the accuracy deteriorates because the distribution assumptions are often unrealistic and violated in practice
Solution Approach 1:
The patent changes the parameter basis for threshold determination from theoretical distribution parameters (mean, standard deviation) to empirical parameters derived directly from the data (quantiles of sorted error values). This allows the threshold to adapt to the actual data distribution without relying on unrealistic assumptions, thereby maintaining systematic determination while improving accuracy
Solution Approach 2:
The patent uses a simple sorting operation and quantile selection instead of complex distribution fitting procedures. This disposable, computationally inexpensive approach replaces expensive and assumption-heavy parametric methods, achieving both ease of implementation and improved accuracy
2Adaptability or versatility
If non-parametric methods are used to avoid distribution assumptions, then the flexibility improves, but the threshold determination becomes ad-hoc and requires significant tuning efforts
Solution Approach 1:
The patent makes the system self-service by automatically determining the threshold through a systematic procedure: calculate prediction errors, sort them, and select the appropriate quantile. This eliminates the need for manual tuning while maintaining the flexibility of non-parametric methods, resolving the contradiction between adaptability and complexity
3Reliability
If machine learning methods are used for anomaly detection, then the detection capability improves, but the requirement for labeled training data increases, which is expensive and requires expert knowledge
Solution Approach 1:
The patent extracts only the essential information needed for anomaly detection (prediction errors from a trained model) and discards the need for expensive labeled anomaly data. By focusing on the deviation between predicted and actual values, the method achieves reliable anomaly detection without requiring substantial labeled training data
Solution Approach 2:
The patent replaces expensive, hard-to-obtain labeled anomaly data with inexpensive, easily obtained unlabeled data and prediction errors. This disposable approach to data acquisition maintains detection capability while eliminating the burden of collecting and labeling extensive training data
4Device complexity
If statistical models are used for univariate datasets, then the model simplicity is maintained, but the ability to detect correlations between multiple metrics is lost
Solution Approach 1:
The patent segments the multivariate anomaly detection problem into two parts: first, use a simple univariate statistical model to predict each metric; second, analyze the prediction errors to detect anomalies. This segmentation allows simple models to be combined effectively, maintaining individual model simplicity while achieving multivariate correlation detection through error analysis
Data Source
AI summary
According to an embodiment of the present invention, an approach accurately detects anomalies or outliers of a time-series dataset. A method for identifying whether a particular data element of the time-series dataset is an outlier comprises predicting a value for that particular data element and obtaining a threshold value that defines, relative to the predicted value, whether an actual value of the data element is an outlier. In an aspect of a present invention embodiment, the threshold value is generated based on historic error values associated with data elements temporally preceding the particular data element of the time-series dataset.