AI Model Evaluation Metrics for Imbalanced Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing evaluation techniques for AI models trained on imbalanced data provide misleading accuracy metrics, leading to biased models that favor the majority class, and fail to accurately evaluate performance in multi-label emotion recognition settings.
Innovation Solution
A device and method that generate evaluation metrics using Cross F1 and Macro F1 scores, which balance performance between positive and negative classes, and select the best performing AI model for deployment based on these metrics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing evaluation techniques are used for AI models trained on imbalanced data, then the evaluation process is simple, but the accuracy metrics are misleading and the model performance on minority classes is poor
Solution Approach 1:
The patent changes the evaluation parameters from traditional accuracy metrics to class-balanced metrics (weighted precision, recall, and F1-score). This involves calculating metrics separately for each class and then computing a weighted average that accounts for the imbalance ratio, thereby providing a more accurate assessment of model performance on minority classes while maintaining reliability across all classes.
2Productivity
If AI models are trained on imbalanced data to achieve high overall accuracy, then training is efficient and simple, but the model becomes biased towards the majority class
Solution Approach 1:
The patent applies parameter changes by modifying the loss function and evaluation metrics to account for class imbalance. The weighted loss function assigns higher penalties to misclassifications of minority classes, while the class-balanced evaluation metrics (weighted precision, recall, F1-score) provide feedback that guides the model to reduce bias towards majority classes during training.
Solution Approach 2:
The patent implements feedback mechanisms through class-balanced evaluation metrics that are computed during training. These metrics provide continuous feedback about the model's performance on each class, allowing the training process to adjust and reduce bias towards majority classes while maintaining overall training efficiency.
3Device complexity
If traditional accuracy metrics are used to evaluate AI models, then the evaluation method is simple, but it fails to detect poor performance on rare but critical events
Solution Approach 1:
The patent changes the evaluation parameters from simple accuracy to class-balanced metrics including weighted precision, recall, and F1-score. These metrics are particularly sensitive to performance on minority classes, enabling reliable detection of critical events while maintaining manageable evaluation complexity through systematic computation formulas.
4Ease of operation
If AI models are evaluated using overall accuracy on imbalanced data, then the evaluation process is straightforward, but it rewards biased models that favor the majority class
Solution Approach 1:
The patent changes the evaluation parameters from overall accuracy to class-balanced metrics (weighted precision, recall, and F1-score). These metrics are computed using systematic formulas that weight each class's performance according to its representation in the dataset, providing precise model performance assessment while maintaining ease of operation through automated calculation.
Data Source
AI summary
A method for controlling an artificial intelligence (AI) device can include obtaining, via a processor in the AI device, an AI model trained on a dataset that includes a majority class and at least one minority class, generating, via the processor, at least one evaluation metric for the AI model based multiplying a first score for positive samples of a target class within the dataset by a number of negative samples of the target class within the dataset and multiplying a second score for the negative samples within the dataset by a number of the positive samples, and outputting, via an output unit in the AI device, the at least one evaluation metric. Also, the method can further include adding the trained AI model to a pool of trained AI models and selecting a best AI model from the pool for deployment based on the at least one evaluation metric.