Ensemble Anomaly Detection With Combined Score Thresholding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing unsupervised anomaly detection methods, such as clustering, principal components analysis, and one-class SVM, often fail to accurately identify anomalous data, leading to missed detections in applications like fraud detection and network security, due to oversimplification of the task.
Innovation Solution
Applying a plurality of anomaly detection models to a dataset, followed by model-free evaluations using methods like Excess-Mass curves, and determining a combined score threshold through techniques like line search to enhance detection accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple anomaly detection models are applied to a dataset, then detection accuracy improves, but computational complexity increases
Solution Approach 1:
The system segments the anomaly detection task by applying multiple specialized detection models (clustering, principal components analysis, one-class SVM, autoencoders) to different aspects of the data, then combines their results through an ensemble approach. This segmentation allows each model to focus on specific patterns while collectively achieving higher detection accuracy.
Solution Approach 2:
The system merges the detection results from multiple independent anomaly detection models into a unified assessment. By combining the strengths of different algorithms through ensemble methods, the system achieves more accurate anomaly detection than any single model could provide alone.
2Measurement precision
If ensemble methods using multiple models are used, then detection accuracy improves, but processing time increases
Solution Approach 1:
The system performs preliminary actions by pre-processing the dataset and training multiple detection models in advance. This preparation work is done before actual anomaly detection is needed, so that when detection is required, the models are already ready to process new data efficiently, reducing real-time processing delays.
Solution Approach 2:
The system applies partial action by selectively applying different detection models based on the specific characteristics of the data and anomaly types being detected. Rather than always running all possible models, the system adjusts the ensemble composition to balance accuracy requirements with processing time constraints.
Data Source
AI summary
Detecting anomalous data by applying a plurality of models to a data set to yield detection results including anomalous data, applying evaluation methods to the detection results for each of the plurality of models, determining a combined score for the detection results according to the evaluation methods, determining a combined score threshold, and defining a set of detected anomalies according to the combined score and the combined score threshold.