Anomaly Score Normalization Using EVT for Reliable Thresholding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly detection methods rely on subjective threshold selection and presumed statistical distributions, leading to inaccurate anomaly detection due to false positives and negatives, especially in rare event scenarios, and lack flexibility in handling evolving threats.
Innovation Solution
Anomaly detection using extreme value theory (EVT) for normalizing and thresholding anomaly scores without assuming a distribution, employing a peak over threshold (POT) technique to isolate extreme events and a generalized Pareto distribution (GPD) for tail modeling, combined with autoencoders for feature extraction and normalization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If predefined thresholds are subjectively selected by human experts, then anomaly detection can be performed, but detection accuracy deteriorates due to incorrect threshold selection
Solution Approach 1:
The system performs self-calibration by automatically selecting thresholds based on anomaly scores from validation data, eliminating the need for manual expert intervention. The threshold is derived objectively from the data distribution rather than subjective human judgment.
Solution Approach 2:
The threshold selection is performed in advance during system initialization or calibration phase using validation data. This preliminary action establishes the threshold before actual anomaly detection begins, ensuring consistent and accurate detection without requiring real-time manual adjustment.
2Reliability
If anomaly threshold is set too high, then false negatives increase, but if set too low, then false positives increase
Solution Approach 1:
The system uses validation data to evaluate different threshold settings and their impact on detection performance. By measuring false positives and false negatives on validation data, the system receives feedback to objectively determine the optimal threshold that balances both error types.
Solution Approach 2:
The threshold parameter is optimized by analyzing the distribution of anomaly scores on validation data. The system identifies the threshold value that best separates normal from anomalous cases based on empirical evidence rather than arbitrary selection.
3Measurement precision
If presumed statistical distributions are used for non-scoring approaches, then anomaly detection can be performed, but accuracy deteriorates due to inadequacy of distributional assumptions
Solution Approach 1:
The invention extracts and removes the problematic element of distributional assumptions from the anomaly detection process. By using anomaly scores directly with empirically determined thresholds, the method eliminates the need to presume Gaussian or other statistical distributions that may not fit the actual data.
Solution Approach 2:
The patent replaces the theoretical statistical mechanics approach (presumed distributions) with an empirical data-driven approach. Instead of relying on mathematical models of data generation, the system uses actual anomaly score distributions from validation data to determine thresholds.
4Measurement precision
If Gaussian distribution models are used, then simple anomaly detection can be performed, but accurate prediction of tail regions is not possible because extreme data points are rare
Solution Approach 1:
The system performs preliminary analysis of anomaly scores on validation data to understand the actual distribution characteristics, including tail behavior. This preliminary action captures empirical evidence about extreme values before deploying the detection system, allowing accurate threshold setting even when extreme events are rare in the training data.
Solution Approach 2:
The method changes from using fixed theoretical distribution parameters (Gaussian) to empirically determined parameters based on validation data. By analyzing the actual anomaly score distribution including tail regions in validation data, the system adapts its threshold to reflect real extreme event characteristics.
Data Source
AI summary
The present invention relates to threshold estimation and calibration for anomaly detection. Herein are machine learning (ML) and extreme value theory (EVT) techniques for normalizing and thresholding anomaly scores without presuming a values distribution. In an embodiment, a computer receives many unnormalized anomaly scores and, according to peak over threshold (POT), selects a highest subset of the unnormalized anomaly scores that exceed a tail threshold. Based on the highest subset of the unnormalized anomaly scores, parameters of a probability density function are trained according to EVT. After training and in a production environment, a normalized anomaly score is generated based on an unnormalized anomaly score and the trained parameters of the probability density function. Anomaly detection compares the normalized anomaly score to an optimized anomaly threshold.