Anomaly Detection via Critical and Independent Metric Monitoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly detection techniques face scalability challenges with large-scale or streaming data, particularly in terms of processing efficiency, memory requirements, and real-time responsiveness, especially for time-series data in IT operations management (ITOM) systems.
Innovation Solution
Implementing a method that filters out non-critical and correlated metric data streams using generative artificial intelligence (GenAI) to reduce the number of metrics processed, grouping correlated metrics into subsets and monitoring a single representative metric per group, thereby reducing processing load and computational overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If anomaly detection is performed on all metric data streams, then detection completeness is improved, but processing efficiency deteriorates
Solution Approach 1:
The patent extracts and removes non-critical and redundant metric data streams from the monitoring set before anomaly detection. By identifying and eliminating metrics that do not contribute significantly to anomaly detection (through criticality assessment and correlation analysis), the system reduces the volume of data to be processed while preserving detection effectiveness for critical metrics.
Solution Approach 2:
The patent segments the metric data streams into different groups based on their criticality levels and correlation relationships. By dividing the monolithic set of all metrics into critical and non-critical segments, and further grouping correlated metrics together, the system can apply anomaly detection selectively to critical segments, improving processing efficiency without sacrificing detection completeness for important anomalies.
2Measurement precision
If all metric data streams are monitored, then anomaly detection accuracy is improved, but memory requirements increase
Solution Approach 1:
The patent extracts and removes redundant metric data streams that consume memory but contribute minimally to anomaly detection accuracy. By eliminating non-critical and highly correlated metrics, the system reduces the memory footprint required for storing and processing metric data while maintaining detection accuracy for critical metrics.
3Reliability
If comprehensive metric monitoring is performed, then detection coverage is improved, but computational overhead increases
Solution Approach 1:
The patent extracts and eliminates redundant computations by removing non-critical and correlated metric data streams from the monitoring pipeline. This reduces the computational overhead associated with processing, storing, and analyzing metric data while preserving detection coverage for critical metrics through targeted anomaly detection.
Solution Approach 2:
The patent segments metrics into critical and non-critical groups, and further divides critical metrics into correlated groups. By organizing metrics this way, the system reduces computational overhead by applying anomaly detection algorithms only to representative metrics from each correlated group rather than processing every metric individually, while maintaining comprehensive coverage through the hierarchical structure.
4Speed
If real-time anomaly detection is performed on all metrics, then responsiveness is improved, but processing efficiency deteriorates
Solution Approach 1:
The patent extracts and removes non-critical metric data streams before real-time anomaly detection processing. By eliminating metrics that do not require immediate monitoring, the system reduces the real-time processing load and can maintain responsiveness for critical metrics without the burden of processing all metrics at full speed.
Solution Approach 2:
The patent segments metrics by criticality and correlation, enabling differentiated processing where critical metrics receive real-time anomaly detection while non-critical metrics are processed less frequently or not at all. This segmentation allows the system to maintain real-time responsiveness for important metrics while improving overall processing efficiency through selective monitoring.
Data Source
AI summary
In the present application, improved techniques for anomaly detection are disclosed. A plurality of metric data streams is obtained. A first subset of the plurality of metric data streams is identified based on determining that each of the first subset of the plurality of metric data streams satisfies a monitoring criticality criterion. A second subset of the plurality of metric data streams is identified from the first subset of the plurality of metric data streams based on determining that each of the second subset of metric data streams satisfies a metric independence criterion. Anomaly detection is performed with respect to the second subset of the plurality of metric data streams.