Root Cause Analysis in Multivariate Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current anomaly detection systems, particularly those using univariate approaches, face limitations in accurately identifying multivariate anomalies and determining their root causes due to their inability to consider correlations between features, leading to decreased accuracy and complexity in interpreting results.
Innovation Solution
A method and system that utilize a root cause analyzer to train a classification model on data samples and anomaly labels, employing techniques like multivariate unsupervised anomaly detection and the SHAP algorithm to determine the relative contributions of anomalous features, providing insights into the causes of anomalies and enabling more accurate anomaly detection and root cause analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If univariate anomaly detection algorithms are used to analyze each metric independently, then the system is computationally simpler and easier to interpret, but the accuracy in identifying multivariate anomalies decreases due to inability to consider correlations between features
Solution Approach 1:
The system segments the anomaly detection process into two distinct phases: (1) an unsupervised multivariate detection phase that identifies anomalies by analyzing correlations across multiple features simultaneously, and (2) a supervised classification phase that uses the detected anomalies as training data to create an interpretable classifier. This segmentation allows the system to leverage the strengths of both approaches - the multivariate correlation analysis of unsupervised methods and the interpretability of supervised classification.
Solution Approach 2:
The system introduces an intermediary component - a supervised classification model trained on anomalies detected by the unsupervised multivariate system. This intermediary translates the complex multivariate anomaly detections into a form that provides intrinsic indications of underlying reasons, thereby bridging the gap between accurate multivariate detection and interpretable root cause analysis.
2Loss of information
If supervised anomaly detection techniques are used, then the system provides intrinsic indications regarding underlying reasons for anomalies, but the system requires training data and becomes more complex
Solution Approach 1:
The system performs preliminary unsupervised anomaly detection to identify anomalous data points before training the supervised classifier. This preliminary action creates a dataset of labeled anomalies that can be used to train the supervised model, eliminating the need for manually labeled training data and reducing overall system complexity.
Solution Approach 2:
The system is self-service in that it automatically generates its own training data through the unsupervised anomaly detection phase. The anomalies detected by the multivariate unsupervised system serve as the training labels for the supervised classifier, allowing the system to bootstrap the supervised learning process without external intervention or manual annotation.
3Measurement precision
If multivariate unsupervised anomaly detection is used to identify anomalies by considering correlations between features, then the accuracy of anomaly detection improves, but the ability to determine root causes decreases due to black-box nature of the models
Solution Approach 1:
The system implements a feedback loop where the unsupervised multivariate anomaly detection results are fed into a supervised classification model. The supervised model learns from the patterns in the multivariate anomalies and provides feedback in the form of interpretable predictions with intrinsic indications of underlying reasons, thereby maintaining high detection accuracy while improving root cause interpretability.
Solution Approach 2:
The system changes the parameter representation by transforming the complex multivariate feature space analyzed by the unsupervised system into a simplified classification space handled by the supervised model. This parameter transformation allows the system to maintain the correlation-based detection accuracy while presenting results in an interpretable format that reveals root causes.
Data Source
AI summary
Described embodiments provide systems and methods for anomaly detection and root cause analysis. A root cause analyzer receives a plurality of data samples input to an anomaly detection engine, and a corresponding plurality of anomaly labels output from the anomaly detection engine. The root cause analyzer trains a classification model using the plurality of data samples and the corresponding plurality of anomaly labels. The root cause analyzer determines, using the trained classification model and the plurality of data samples, relative contributions of anomalous features in a data sample of the plurality of data samples, to a prediction that the data sample is anomalous. The root cause analyzer provides the relative contributions of anomalous features to a device, to determine an action in response to the prediction that the data sample is anomalous.