Multivariate Anomaly Detection for Root Cause Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current anomaly detection systems in complex computing applications face challenges due to their inability to effectively monitor interdependent resources, leading to inefficiencies in identifying and addressing anomalies, with statistical anomaly detectors generating false positives and requiring extensive user-labeled data for supervised training.
Innovation Solution
The system employs a hybrid approach that transitions between unsupervised, machine-assisted supervised, and fully supervised training for anomaly detection models, using biased sampling and automatic labeling to reduce false positives and alleviate the need for extensive user input, allowing for nearly instantaneous anomaly detection and classification in multivariate data environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If statistical anomaly detectors are used to monitor computing applications, then anomaly detection capability is provided, but false positives are generated and extensive user-labeled data is required for supervised training
Solution Approach 1:
The system performs unsupervised pre-training on available data to establish baseline anomaly detection capabilities before supervised fine-tuning. This preliminary action allows the model to learn general patterns without requiring extensive labeled data upfront, thereby reducing the training data requirement while maintaining detection accuracy.
Solution Approach 2:
The system dynamically adjusts training parameters including learning rates, batch sizes, and regularization strengths based on data availability and model performance. By changing these parameters adaptively, the system optimizes training efficiency and reduces the amount of labeled data needed while maintaining high anomaly detection accuracy.
2Measurement precision
If supervised training is used for anomaly detection models, then detection accuracy is improved, but extensive user-labeled data is required
Solution Approach 1:
The system performs unsupervised pre-training to establish baseline capabilities before supervised fine-tuning. This preliminary action allows the model to learn general patterns without requiring extensive labeled data upfront, thereby reducing the labeled training data quantity needed while maintaining high detection precision.
Solution Approach 2:
The system implements continuous learning where the model continuously adapts to new patterns in the data stream. This continuous useful action allows the model to maintain high precision with minimal additional labeled data, as it constantly refines its understanding of normal and anomalous patterns from ongoing operations.
3Adaptability or versatility
If complex computing applications with interdependent resources are monitored, then comprehensive monitoring coverage is achieved, but difficulty in identifying root cause anomalies increases
Solution Approach 1:
The system segments the complex computing application into individual resources and their interdependencies, creating separate monitoring models for each component. This segmentation allows the system to track how anomalies propagate through the system, making it easier to identify the root cause by tracing anomaly origins through the segmented resource graph rather than analyzing the entire system as a monolith.
Solution Approach 2:
The system introduces an intermediary anomaly propagation model that captures the relationships and dependencies between resources. This intermediary layer translates complex interdependencies into interpretable anomaly flow patterns, enabling the system to maintain comprehensive monitoring coverage while simplifying root cause identification by showing how anomalies propagate through intermediate resources.
4Productivity
If traditional anomaly detection systems are deployed, then basic anomaly detection is provided, but false positives occur and response prioritization is difficult
Solution Approach 1:
The system implements feedback loops where detected anomalies are evaluated and used to refine the model's understanding of what constitutes true anomalies versus false positives. This feedback mechanism continuously improves the false positive rate by learning from past detections and adjusting detection thresholds and patterns, thereby improving reliability while maintaining high detection efficiency.
Solution Approach 2:
The system combines multiple anomaly detection approaches including unsupervised learning, supervised fine-tuning, and rule-based detection into a composite detection framework. This composite approach leverages the strengths of each method while compensating for their individual weaknesses, resulting in reduced false positives and improved ability to prioritize true anomalies for response.
Data Source
AI summary
Techniques are disclosed for summarizing, diagnosing, and correcting the cause of anomalous behavior in computing systems. In some embodiments, a system identifies a plurality of time series that track different metrics over time for a set of one or more computing resources. The system detects a first set of anomalies in a first time series that tracks a first metric and assigns a different respective range of time to each anomaly. The system determines whether the respective range of time assigned to an anomaly overlaps with timestamps or ranges of time associated with anomalies from one or more other time series. The system generates at least one cluster that groups metrics based on how many anomalies have respective ranges of time and/or timestamps that overlap. The system may preform, based on the cluster, one or more automated actions for diagnosing or correcting a cause of anomalous behavior.