Semi-supervised Anomaly Detection Framework for Purpose-Oriented Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning-based anomaly detection systems face challenges with concept drift, leading to false positives, and lack purpose-oriented functionality, resulting in irrelevant anomaly identification.
Innovation Solution
A semi-supervised anomaly detection framework that integrates manual grading to train a supervised classification model on top of an unsupervised anomaly detection model, addressing concept drift and improving relevance by calculating statistical uncertainty and selecting graded samples effectively.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If unsupervised anomaly detection models are used to detect anomalies, then the system can identify statistical outliers, but the system produces false positives due to concept drift and lacks purpose-oriented functionality
Solution Approach 1:
A semi-supervised classification model is introduced as an intermediary component between the unsupervised anomaly detection model and the final anomaly reporting. This intermediary model processes the anomaly scores from the unsupervised model and applies purpose-oriented classification to filter out false positives, thereby improving reliability while reducing harmful false positive outputs
Solution Approach 2:
Manual grading of anomalies is implemented as a feedback mechanism where human experts review and label anomaly samples. This feedback is used to continuously train and refine the semi-supervised classification model, enabling the system to learn from actual anomaly patterns and reduce false positives over time while maintaining purpose-oriented detection
2Reliability
If manual grading is used to train supervised models, then purpose-oriented anomaly identification is achieved, but the process is time-consuming and requires significant human effort
Solution Approach 1:
Instead of requiring complete manual grading of all anomaly samples, the system uses partial action by selectively grading only a subset of samples. The semi-supervised classification model leverages this partial labeled data to achieve purpose-oriented detection, significantly reducing the time and human effort required compared to full manual grading while maintaining detection accuracy
Solution Approach 2:
The unsupervised anomaly detection model performs preliminary action by automatically identifying potential anomaly samples and generating anomaly scores before human grading is applied. This preliminary processing filters and prioritizes samples for manual grading, reducing the overall time required by focusing human effort only on the most suspicious cases
3Adaptability or versatility
If the unsupervised model is retrained frequently to address concept drift, then adaptation to changing data distributions is improved, but system complexity and operational disruption increase
Solution Approach 1:
The system segments the adaptation process into two independent components: the unsupervised anomaly detection model that handles statistical outlier detection, and the semi-supervised classification model that handles purpose-oriented filtering. This segmentation allows each model to be trained independently on different data types, reducing the complexity and disruption of retraining the entire system when concept drift occurs
Data Source
AI summary
Techniques for implementing a semi-supervised framework for purpose-oriented anomaly detection are provided. In one technique, a data item in inputted into an unsupervised anomaly detection model, which generates first output. Based on the first output, it is determined whether the data item represents an anomaly. In response to determining that the data item represents an anomaly, the data item is inputted into a supervised classification model, which generates second output that indicates whether the data item is unknown. In response to determining that the data item is unknown, a training instance is generated based on the data item. The supervised classification model is updated based on the training instance.