Anomaly Detection Using Surrogate Data Sets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly detection systems in distributed computing landscapes face challenges in efficiently detecting and classifying anomalies due to the rarity of anomalies and the need for large amounts of labeled data, which is expensive and impractical to obtain, and struggle with determining appropriate threshold values for anomaly classification.
Innovation Solution
The system generates surrogate data sets without anomalies, using generative models and Fourier transformations to create data with similar spectral characteristics, allowing for unsupervised learning and dynamic threshold determination to identify anomalies without prior labeling, and facilitates root cause analysis and enriched alerting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If supervised learning with labeled data is used for anomaly detection, then classification precision and recall are improved, but data acquisition cost and complexity increase significantly
Solution Approach 1:
The patent creates surrogate data sets that copy the statistical and spectral characteristics of normal system data without requiring actual anomaly labels. By generating synthetic normal data through sampling and spectral transformation, the system avoids the need for expensive expert-labeled anomaly data while maintaining detection accuracy
Solution Approach 2:
The system performs self-service by automatically determining thresholds through unsupervised learning on surrogate data without requiring external expert labeling. The anomaly detection model trains itself using only normal data characteristics, eliminating dependency on manual anomaly annotation
2Measurement precision
If more labeled anomaly data is collected, then anomaly detection accuracy is improved, but time and resource requirements increase
Solution Approach 1:
The patent performs preliminary action by pre-generating surrogate data sets that capture normal system behavior patterns before actual anomaly detection is needed. These pre-prepared surrogate data sets enable rapid threshold determination without requiring time-consuming collection and labeling of anomaly data
Solution Approach 2:
Instead of collecting actual anomaly data over time, the system copies normal data characteristics into surrogate data sets that can be repeatedly used for threshold training, eliminating the need for lengthy anomaly data acquisition periods
3Ease of operation
If fixed thresholds are used for anomaly classification, then system simplicity is maintained, but adaptability to changing system conditions deteriorates
Solution Approach 1:
The patent implements dynamics by making thresholds adaptive rather than fixed. The system dynamically determines thresholds by training on surrogate data sets that reflect current normal behavior patterns, allowing automatic adaptation to system changes while maintaining simple binary classification output
Solution Approach 2:
The system changes parameters by adjusting thresholds based on learned distributions from surrogate data. Instead of using static threshold values, the system modifies thresholds dynamically according to the statistical characteristics of the surrogate data, enabling adaptability without complex classification logic
4Device complexity
If unsupervised learning is used for anomaly detection, then labeled data requirements are reduced, but threshold determination difficulty increases
Solution Approach 1:
The patent introduces an intermediary solution by using surrogate data sets as a bridge between unsupervised learning and threshold determination. The surrogate data act as a mediator that translates normal system characteristics into actionable threshold values, making threshold determination tractable without labeled anomaly data
Solution Approach 2:
The system performs preliminary threshold determination by training on surrogate data sets that capture normal behavior patterns. This pre-computation of thresholds based on synthetic normal data eliminates the difficulty of determining thresholds from raw unsupervised data alone
Data Source
AI summary
Systems and methods include determination of training data instances associated with a respective time periods based on time-series data of each of several metrics, training of a score generator, based on the training data instances, to generate an outlier score, generation of surrogate time-series data of each of the metrics based on the time-series data of each of the metrics, determination of input data instances associated with each one of the respective time periods based on the surrogate time-series data, input of the input data instances to the trained score generator to generate an outlier score for each input data instance, determination of a threshold based on the outlier scores, identification of ones of the training data instances associated with an outlier score greater than the threshold, and identification of an anomaly associated with each of the training data instances associated with an outlier score greater than the threshold.