Anomaly Detection System Using Z-Score Transition Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The vast amount of data collected from computer systems and other sources often contains anomalies that are difficult for human experts to detect and analyze due to their sheer volume, leading to many anomalies being archived or ignored, which can result in missed opportunities for investigation and resolution.
Innovation Solution
A method and system for automatic anomaly detection in data, specifically identifying anomaly transition points using statistical metrics like z-scores and LRZ convolution processes, which generate human-readable explanations to assist domain experts in prioritizing and addressing potential issues.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If human experts manually analyze collected data, then anomaly detection accuracy is improved, but analysis time and resource requirements increase significantly
Solution Approach 1:
The patent segments the data analysis task into automated preprocessing steps (filtering, aggregation) performed by the system, and selective human review of only suspicious anomalies. This division allows human experts to focus their analytical skills on critical cases while the automated system handles routine processing, resolving the contradiction between detection accuracy and analysis time.
Solution Approach 2:
The patent introduces an automated anomaly detection system as an intermediary between raw data collection and human expert analysis. This intermediary performs initial filtering, scoring, and prioritization of anomalies, presenting only the most significant cases to human experts. This mediator role maintains high detection accuracy while dramatically reducing the time human experts must spend on analysis.
2Loss of information
If all collected data is archived and retained, then data completeness is improved, but storage requirements and processing complexity increase
Solution Approach 1:
The patent extracts and retains only the most significant anomalies and relevant data points for further analysis, rather than archiving all collected data. By applying automated filtering and anomaly scoring, the system identifies and extracts only those data elements that require human review, reducing storage requirements and processing complexity while maintaining data completeness for critical cases.
Solution Approach 2:
The patent applies different data retention and processing qualities to different data elements based on their anomaly scores and relevance. High-scoring anomalies receive full archival and detailed processing, while low-scoring normal data receives minimal processing. This local differentiation of data quality treatment reduces overall system complexity while preserving completeness where it matters most.
3Productivity
If automated anomaly detection is implemented, then analysis speed is improved, but false positive rate increases
Solution Approach 1:
The patent implements feedback loops where human expert review of automated anomaly detections provides corrective feedback to refine the detection algorithm. By continuously learning from human verification of false positives and true anomalies, the system improves its detection accuracy over time while maintaining high analysis speed. This feedback mechanism resolves the contradiction by allowing rapid automated screening while progressively reducing false positives through adaptive learning.
Data Source
AI summary
A method and system for detecting anomaly transition point candidates in performance metadata. The method can be applied to computer system performance monitoring. Anomaly candidates, indicative of a possible transition, of a process generating the performance metadata, to or from an anomalous behavior mode are identified, for example by comparing z-scores to the left and right of various timestamps and identifying anomaly candidates when the z-scores are significantly different. Anomaly candidates occur singularly rather than as pairs of endpoints of an anomaly interval. For at least one of the anomaly candidates, an explanatory predicate, indicative of a human-readable explanation of behavior of the process, can be generated. The set of anomalies can then be filtered, for example by removing those without explanatory predicates or replacing clusters of anomalies with a most relevant anomaly.