Extreme Value Theory Outlier Scoring for Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for detecting anomalies in engineering systems, such as virtual storage and IoT devices, using histogram-based outlier scores (HBOS) face inaccuracies due to binning limitations, where rare events are misclassified with uniform outlier scores across bins, leading to incorrect anomaly identification.
Innovation Solution
Implementing extreme value theory (EVT) to calculate individual outlier scores for each data feature by determining parameters, calculating probability values, and assigning risk factor scores based on these probabilities, eliminating the need for binning and enhancing accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If histogram-based outlier scoring (HBOS) is used to group features into bins, then processing speed is maintained, but measurement precision deteriorates due to uniform outlier scores assigned to all values within each bin
Solution Approach 1:
The patent extracts the problematic binning step from the anomaly detection process and replaces it with direct EVT-based probability calculation for each individual data point. This removes the source of measurement precision deterioration while maintaining the overall processing efficiency through vectorized operations.
Solution Approach 2:
The patent changes the fundamental parameter from binned histogram counts to continuous extreme value distribution parameters (shape parameter ξ and scale parameter σ). This transformation allows for precise outlier scoring of individual points while maintaining computational efficiency through parameter-based modeling rather than discrete binning.
2Device complexity
If binning is used to accommodate rare events, then device complexity is reduced, but measurement precision deteriorates due to widened bins causing uniform scoring of dissimilar values
Solution Approach 1:
The patent removes the binning mechanism entirely and replaces it with direct application of extreme value theory to each data point. This eliminates the need to widen bins for rare events while maintaining algorithmic simplicity through the use of established EVT distributions.
Solution Approach 2:
The patent applies local quality by allowing each data point to have its own precise outlier score based on its specific position in the distribution, rather than forcing all points in a widened bin to share the same score. This enables fine-grained differentiation of rare events without increasing overall system complexity.
Data Source
AI summary
A subset of data that includes a feature may be selected from a dataset. Parameters from the selected subset of data are determined and an extreme value theory (EVT) algorithm is implemented to determine a probability value for the feature based at least in part on the determined parameters. Based on the determined probability value for the feature, an outlier score is generated for the feature. Based on the outlier score being above a threshold, the subset is identified as anomalous.