Balanced Multimodal Time-Series Data for Low False Positives
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly detection methods, particularly in cloud infrastructure, suffer from class imbalance due to the rarity of anomalies and lack of effective evaluation metrics, leading to high false positive detections and inefficient resource utilization.
Innovation Solution
A supervised learning pipeline is developed that includes a labeling scheme, sample generation and balancing method, and task-specific evaluation metric to create a balanced training dataset, using system tickets to label each timestep and predict anomalies before they occur, employing a model with a one-dimensional convolutional layer and dense layer for accurate detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If unsupervised anomaly detection methods are used, then the system can operate without labeled data, but the detection accuracy is low and false positive rate is high
Solution Approach 1:
The patent applies preliminary action by pre-labeling historical timeseries data with anomaly labels before training the model. This preprocessing step creates a supervised training dataset that enables the model to learn from known anomalies, thereby improving detection accuracy while maintaining the ability to work with unlabeled operational data during deployment
2Device complexity
If traditional threshold detection methods are used, then the implementation is simple, but the false positive detections increase and resource utilization becomes inefficient
Solution Approach 1:
The patent replaces mechanical threshold-based detection with a machine learning model that learns complex patterns from historical data. The supervised model substitutes simple threshold comparisons with intelligent pattern recognition, reducing false positives and improving resource utilization by accurately distinguishing true anomalies from normal variations
3Measurement precision
If supervised learning is applied, then detection accuracy improves, but the requirement for labeled data increases which is typically unavailable
Solution Approach 1:
The patent applies preliminary action by pre-labeling historical timeseries data with anomaly labels before training the model. This preprocessing step creates a supervised training dataset that enables the model to learn from known anomalies, thereby improving detection accuracy while maintaining the ability to work with unlabeled operational data during deployment
Solution Approach 2:
The patent uses historical timeseries data as a copy or representation of normal operational patterns. By training on labeled historical data and then applying the model to unlabeled operational data, the system copies the learning from past patterns to future predictions without requiring continuous labeled data
Data Source
AI summary
One or more computer processors labeling each timestep comprised in historical multivariate timeseries data logged from a plurality of systems. The one or more computer processors split each labeled timestep into a plurality of training sets, wherein each training set does not overlap with each remaining training set in the plurality of training sets. The one or more computer processors train a supervised model with the plurality of training sets, wherein the supervised model comprises a one dimensional convolutional layer, a one dimensional max pooling layer, and a dense layer. The one or more computer processors detect one or more anomalous timesteps within the new multivariate timeseries data utilizing the train supervised model. The one or more computer processors remediate one or more systems associated with the one or more anomalous timesteps.