Synthetic Anomaly Quality Scoring With Bayesian Density
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data quality metrics fail to quantify the quality of synthetic anomalies, leading to poor performance in anomaly detection models due to the inclusion of unrealistic or indistinguishable anomalies, especially when real anomalous examples are scarce and biased.
Innovation Solution
A novel metric that estimates the quality of synthetic anomalies by capturing an anomaly detector's uncertainty, considering both model prediction and data density, using a Bayesian approach to determine class-conditional probability and data density, enabling efficient evaluation without retraining.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If synthetic anomalies are used to improve model training, then the model can be trained with limited real anomalies, but the synthetic anomalies may be unrealistic or indistinguishable from normal samples, deteriorating detector performance
Solution Approach 1:
The patent replaces manual evaluation of synthetic anomaly quality with an automated anomaly detector system. The detector uses machine learning algorithms to automatically assess whether synthetic anomalies are realistic and distinguishable from normal samples, substituting human judgment with computational evaluation.
Solution Approach 2:
The patent implements a feedback loop where the anomaly detector evaluates synthetic anomalies, provides quality scores, and this information feeds back into the anomaly generation process. This allows continuous improvement of synthetic anomaly quality based on detector performance metrics.
2Measurement precision
If traditional data quality metrics are used to evaluate training examples, then the impact on model performance can be measured, but these metrics cannot quantify the quality of external anomalies
Solution Approach 1:
The patent creates a universal evaluation framework that can assess both training examples and external anomalies using the same anomaly detector. The detector serves multiple functions: training normal samples, evaluating synthetic anomalies, and providing quality metrics, making the system adaptable to different evaluation scenarios.
Solution Approach 2:
The patent changes the evaluation parameters from traditional data quality metrics to anomaly-specific metrics. Instead of measuring general data quality, the system uses anomaly scores, detection confidence, and distinguishability metrics that are specifically tailored for evaluating anomaly samples.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method of labeling data for machine learning models wherein the method includes generating one or more synthetic datasets that includes one or more labels and one or more anomalies associated with a training dataset; determining, utilizing an anomaly detector, an estimate associated with a class-conditional probability of the one or more synthetic datasets in response to mapping anomaly scores assigned by the anomaly detector, determining, utilizing a data density estimator, a data density associated with the one or more synthetic datasets in response to a function of a rarity score or a kernel density estimation function associated with the training dataset, determining a quality score associated with the one or more synthetic datasets utilizing at least the data density and an estimate associated with class-conditional probability, and outputting the quality score associated an anomaly of the one or more synthetic datasets.