Anomaly Detection Ensemble for IT Infrastructure Time Series
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly detection techniques face challenges in accurately identifying anomalies in complex, multi-modal IT infrastructure data streams that do not exhibit stationarity, often resulting in high false positives or requiring significant labeled training data, which becomes outdated quickly.
Innovation Solution
An automated anomaly detection system using an ensemble of deep learning models, specifically LSTM models with autoencoders, uncertainty estimation, and dropouts, trained on clustered time series data, which aggregates and identifies anomalies based on majority voting and groups them by relationships for effective mitigation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing anomaly detection techniques are used on complex multi-modal IT infrastructure data streams, then the system can process the data, but the accuracy of anomaly identification deteriorates due to high false positives and inability to handle non-stationary data
Solution Approach 1:
The patent segments the complex multi-modal data streams into multiple univariate time series components, each representing a specific metric or aspect of the IT infrastructure. This segmentation allows each component to be analyzed independently by specialized anomaly detection models, improving overall detection accuracy while reducing false positives caused by trying to analyze complex multi-modal data as a whole.
Solution Approach 2:
The patent transforms the complex multi-modal data into multiple univariate time series representations, changing the parameter structure from a high-dimensional multi-modal format to multiple independent univariate formats. This parameter transformation enables the use of specialized univariate anomaly detection models that are more accurate for each specific type of metric, thereby improving overall anomaly identification accuracy.
2Adaptability or versatility
If labeled training data is used to train anomaly detection models, then the models can be trained, but the training data becomes outdated quickly requiring significant ongoing labeled data collection and model retraining
Solution Approach 1:
The patent implements self-service anomaly detection by combining unsupervised learning approaches with ensemble methods that do not require labeled training data. The system automatically adapts to changing data patterns through the ensemble of multiple specialized models that can detect anomalies in non-stationary data without needing manual retraining with newly labeled data, thereby eliminating the time loss associated with continuous data labeling and model retraining.
Solution Approach 2:
The patent merges multiple specialized univariate anomaly detection models into an ensemble system that collectively handles multi-modal data streams. This combination allows the system to maintain adaptability to changing patterns across different data types without requiring centralized retraining, as each model independently adapts to its specific data stream while the ensemble integrates their findings.
3Measurement precision
If multiple specialized anomaly detection models are deployed for different data streams, then detection accuracy improves, but the system complexity and computational resources required increase
Solution Approach 1:
The patent segments the anomaly detection task into multiple specialized univariate models, each handling a specific data stream or metric type. While this increases the number of models, it reduces the complexity of each individual model by focusing on single-variable patterns, making them computationally efficient and easier to maintain despite the increased overall system composition.
Solution Approach 2:
The patent creates a universal ensemble framework that can accommodate multiple specialized univariate anomaly detection models for different data streams. This multi-functional architecture allows the same ensemble structure to handle various types of IT infrastructure data (CPU metrics, memory metrics, network metrics, etc.) uniformly, reducing the need for separate complex systems for each data type and thereby managing overall system complexity.
Data Source
AI summary
Described are techniques for automated anomaly detection including a technique comprising training an ensemble of deep learning models using clustered time series training data from numerous components in an Information Technology (IT) infrastructure. The technique further comprises inputting aggregated time series data to the ensemble of deep learning models and identifying anomalies in the aggregated time series data based on respective portions of the aggregated time series data that are indicated as anomalous by a majority of deep learning models in the ensemble of deep learning models. The technique further comprises grouping the anomalies according to relationships between the anomalies and performing a mitigation action in response to grouping the anomalies.