Unsupervised ML Model Evaluation via Benchmarking and Mean Average Precision
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Evaluating and comparing the performance of unsupervised or semi-supervised machine learning models for anomaly detection is challenging due to the rarity of outlier events and the lack of labeled datasets, making it difficult to assess their effectiveness in detecting multiple types of anomalies.
Innovation Solution
A method is introduced that involves generating benchmarking datasets with a larger number of negative samples compared to positive samples for each type of anomalous activity, training and cross-validating machine learning models using these datasets, and computing mean average precision to rank candidate algorithms for deployment in anomaly detection systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If unsupervised or semi-supervised machine learning models are used for anomaly detection, then the system can operate without extensive labeled data, but the evaluation and comparison of model performance becomes difficult due to lack of labeled datasets
Solution Approach 1:
The patent introduces an intermediary evaluation framework that uses a small amount of labeled data combined with unsupervised learning techniques to assess model performance. This intermediary approach bridges the gap between fully supervised evaluation (which requires extensive labeled data) and completely unsupervised operation (which lacks evaluation capability), enabling performance measurement without requiring large labeled datasets for deployment
Solution Approach 2:
The patent performs preliminary evaluation actions by training models on available labeled data and then using cross-validation with synthetic or semi-labeled datasets to assess performance before full deployment. This preliminary evaluation allows the system to compare different unsupervised models using metrics like average precision computed from limited labeled validation sets, enabling informed model selection without extensive labeling
2Adaptability or versatility
If multiple types of anomalies are detected, then the system coverage is improved, but the variability in performance metrics increases making fair comparison difficult
Solution Approach 1:
The patent segments the evaluation process by computing separate average precision metrics for each anomaly type individually, then aggregating these segmented results. This segmentation allows the system to evaluate performance on each anomaly type separately (improving adaptability) while maintaining stable comparison through consistent metric computation and aggregation across all types
Solution Approach 2:
The patent changes the evaluation parameter from raw detection counts to average precision metrics that are normalized and comparable across different anomaly types. By using average precision as the standard parameter and computing it consistently across multiple anomaly types, the system achieves both versatile coverage and stable, fair comparison of model performance
3Reliability
If the number of positive samples is small compared to negative samples, then the system reflects real-world anomaly rarity, but the statistical reliability of performance metrics decreases
Solution Approach 1:
The patent implements feedback through cross-validation where the model's performance on the small positive samples is repeatedly evaluated and averaged. This feedback mechanism accumulates statistical evidence across multiple validation folds, improving the reliability of performance metrics despite the small number of positive samples, while maintaining the realistic class imbalance
Data Source
AI summary
Methods of evaluating and deploying machine learning models for anomaly detection of a monitored system and related systems. Candidate machine learning algorithms are configured for anomaly detection of the monitored system. For each combination of candidate machine learning algorithm with type of anomalous activity, training and cross-validation sets are drawn from a benchmarking dataset. Using each of the training and cross-validation sets, a machine-learning model is trained and validated using the cross-validation set with average precision as a performance metric. A mean average precision value is then computed across these average precision performance metrics. A ranking value is computed for each candidate machine learning algorithm, and a machine learning algorithm is selected from the candidate machine learning algorithms based upon the computed ranking values. The selected machine learning model is deployed to a monitoring system that executes the deployed machine learning model to detect anomalies of the monitored system.