Metric Relationship Graph for Anomaly Detection in Time Series
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current monitoring systems in large-scale computing environments face challenges in detecting anomalies and relationships between metrics due to the manual configuration of rules, limited visual oversight, and high false positive rates, which can lead to missed issues and inefficiencies in problem detection and resolution.
Innovation Solution
A system that automatically identifies relationships between metrics by dividing time series data into windows, calculating hash values, and creating a graph to analyze anomalies, using modules such as a windowing module, hash module, candidate identification module, and metric relationship graph creation module, along with techniques like locality-sensitive hashing and stacked autoencoders to infer dependencies and detect anomalies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual rule configuration is used for anomaly detection, then detection accuracy may be improved through expert knowledge, but the complexity and time consumption of system setup increases significantly
Solution Approach 1:
The system automatically discovers metric relationships and generates detection rules without requiring manual expert configuration. The metric relationship graph is built autonomously by analyzing historical metric data and identifying correlations, allowing the monitoring system to serve itself rather than requiring continuous human intervention for rule creation and maintenance.
Solution Approach 2:
The patent introduces an intermediate representation layer (metric relationship graph) that mediates between raw metric data and anomaly detection logic. This graph structure serves as an intermediary model that captures metric relationships in a standardized format, enabling automated rule generation while maintaining detection accuracy that would otherwise require manual expert configuration.
2Loss of information
If more metrics are monitored simultaneously, then system visibility is improved, but the ability to visually track and analyze all metrics decreases
Solution Approach 1:
The patent segments the large set of monitored metrics into meaningful groups based on their relationships in the metric relationship graph. Instead of presenting all metrics flatly, the system organizes them into clusters or subsets that share relationships, allowing administrators to analyze metrics in manageable groups while maintaining visibility into the complete system state through the graph structure.
Solution Approach 2:
The patent adds a relational dimension to metric monitoring by building a metric relationship graph that shows how metrics connect to each other. This transforms the monitoring view from a flat list of individual metrics to a multi-dimensional structure where metrics are understood in the context of their relationships, enabling better analysis of system-wide patterns without being overwhelmed by the volume of individual metrics.
3Productivity
If automated anomaly detection is implemented, then detection speed is improved, but false positive rates increase
Solution Approach 1:
The system uses the metric relationship graph to provide feedback context for anomaly detection. When an anomaly is detected in one metric, the system can quickly query the graph to identify related metrics and their relationships, allowing for rapid contextual analysis that reduces false positives while maintaining fast detection speeds. The graph structure enables feedback loops where detection results inform further analysis of related metrics.
4Measurement precision
If relationships between all metric pairs are analyzed, then detection accuracy is improved, but computational complexity increases
Solution Approach 1:
The patent performs preliminary analysis to build the metric relationship graph using historical data before actual anomaly detection occurs. This pre-computation of relationships allows the system to store metric correlations in an optimized graph structure, so that during runtime anomaly detection, the system can quickly query pre-established relationships rather than computing them in real-time, significantly reducing computational complexity while maintaining detection accuracy.
Data Source
AI summary
A system includes a windowing module that divides time series data for each metric into portions. Each portion corresponds to a respective window of time. A hash module calculates a hash value for each of the portions for each of the metrics. An identification module compares the hash values for each pair of metrics and, for a selected pair of metrics, counts how many windows of time in which the hash values of the selected pair of metrics are equal. A pair is identified as a candidate pair in response to the count exceeding a threshold. A metric graph module creates a first edge in a graph based on the candidate pair of metrics. Each of the metrics is a node in the graph and direct relationships between each pair of the metrics are edges in the graph. An anomaly combination module analyzes an anomaly condition based on the graph.