Image Recognition Accuracy Evaluation Using Uncertainty Confidence
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods struggle to accurately evaluate the inference accuracy of image recognition models, particularly in environments that have changed or are unseen, as they do not account for the confidence level of the model's predictions, leading to inaccurate assessments of model performance.
Innovation Solution
An information processing method that involves acquiring an inference result and uncertainty information from an image recognition model, generating inference index information with confidence levels indicating reliability, and evaluating model accuracy based on this information, allowing for differentiation between accurate and confident predictions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional evaluation indices (mIoU, mPA) are used to evaluate inference accuracy, then the evaluation process is simple, but the evaluation precision is insufficient and cannot accurately reflect model reliability
Solution Approach 1:
The evaluation process is segmented into multiple dimensions: correctness evaluation (whether prediction matches ground truth) and confidence evaluation (uncertainty measurement). This segmentation allows simultaneous assessment of both accuracy and reliability, resolving the contradiction between simple evaluation and precise evaluation by breaking down the complex evaluation task into manageable components.
Solution Approach 2:
The patent introduces a new evaluation dimension (confidence level/uncertainty) alongside the traditional correctness dimension. By adding this another dimension to the evaluation space, the system can distinguish between correct predictions made with high confidence versus those made with low confidence, thereby improving measurement precision without excessively complicating the evaluation framework.
2Reliability
If retraining is performed to improve model performance in changed environments, then model accuracy can be improved, but time and computational resources are consumed
Solution Approach 1:
The system performs preliminary evaluation using uncertainty measurement before deciding whether retraining is necessary. By assessing model confidence on test data first, the system can identify cases where the model is already sufficiently reliable, avoiding unnecessary retraining operations and thus reducing time loss while maintaining model performance.
Solution Approach 2:
The patent implements a feedback mechanism where uncertainty evaluation results guide the retraining decision-making process. When uncertainty is high or performance degradation is detected, the system triggers retraining; when uncertainty is low and performance is acceptable, retraining is deferred. This feedback loop optimizes the balance between maintaining reliability and minimizing time consumption.
Data Source
AI summary
An evaluation device acquires an inference result of an evaluation target image by an image recognition model generated by machine learning and uncertainty information indicating instability degree of the inference result, generates inference index information in which a confidence level indicating reliability of the inference result is assigned to information indicating whether the inference result is correct or incorrect by using a correct answer label, the inference result, and the uncertainty information associated with the evaluation target image, and evaluates inference accuracy of the image recognition model based on the inference index information.