Hybrid Anomaly Detection Using Unsupervised and Supervised Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current anomaly detection methods in IT infrastructure often struggle with accuracy, particularly in environments with numerous features, where supervised training is tedious and unsupervised models may produce false positives, leading to inefficient identification and classification of anomalies.
Innovation Solution
Combining unsupervised and supervised machine learning models, where the unsupervised model generates initial predictions, and the supervised model, trained with user feedback, refines these predictions, with frequent retraining of the supervised model to improve accuracy and maturity scores.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If unsupervised machine learning models are used for anomaly detection, then the system can operate without labeled training data, but the model produces false positives and lacks accuracy
Solution Approach 1:
The patent combines unsupervised and supervised machine learning models into a hybrid anomaly detection system. The unsupervised model generates initial anomaly predictions without labeled data, while the supervised model refines these predictions using labeled training data to reduce false positives and improve accuracy. This merging allows the system to leverage the advantages of both approaches.
Solution Approach 2:
The patent introduces an intermediary component that bridges the unsupervised and supervised models. The unsupervised model serves as a preliminary filter that identifies potential anomalies, which are then passed to the supervised model for verification and refinement. This intermediary approach allows the supervised model to focus computational resources on refining specific anomaly candidates rather than processing all raw data.
2Measurement precision
If supervised machine learning models are used for anomaly detection, then the model achieves higher accuracy, but the training process becomes tedious and complex
Solution Approach 1:
The patent segments the anomaly detection task into two distinct phases: an unsupervised phase for initial anomaly identification and a supervised phase for refinement. This segmentation allows the complex supervised training to be applied only to specific anomaly candidates rather than the entire dataset, reducing overall training complexity while maintaining high accuracy.
Solution Approach 2:
The patent performs preliminary action by using the unsupervised model to pre-identify anomaly candidates before applying the supervised model. This preliminary filtering reduces the volume of data that requires complex supervised training, making the overall process more manageable while preserving the accuracy benefits of supervised learning.
3Reliability
If frequent retraining of supervised models is performed, then the maturity and accuracy of the system improve, but the computational resources and time required increase
Solution Approach 1:
The patent implements periodic retraining of the supervised model at scheduled intervals rather than continuously. This periodic action allows the system to maintain updated models and improve maturity over time while avoiding the constant computational overhead of continuous retraining. The unsupervised model operates continuously between retraining cycles to maintain anomaly detection capabilities.
Solution Approach 2:
The patent ensures continuity of useful action by having the unsupervised model operate continuously to detect and flag potential anomalies, while the supervised model undergoes periodic retraining to maintain accuracy. This continuous operation of the unsupervised component ensures that anomaly detection never stops, even during supervised model retraining periods.
Data Source
AI summary
A training dataset for anomaly detection is received. An unsupervised machine learning model is trained using at least a portion of the training dataset to generate a trained unsupervised machine learning model. A supervised machine learning model is trained using an output from the unsupervised machine learning model and an anomaly detection feedback associated with the output from the unsupervised machine learning model to generate a trained supervised machine learning model. Both the trained unsupervised machine learning model and the trained supervised machine learning model are provided for combined use in machine learning anomaly detection inference.