Classification Evaluation Support Apparatus for ML Model Stability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Evaluating the accuracy and stability of machine learning models used for multi-class image classification is challenging due to the complexity of calculating matching degrees for multiple classes, making it difficult to assess performance stability.
Innovation Solution
A classification evaluation support apparatus that calculates matching degrees for each class of multiple classes for verification images using a machine learning model, classifies images based on these matching degrees, and displays a separation degree between the ground truth class and other classes, facilitating the evaluation of accuracy and stability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If matching degrees of multiple classes are calculated for each verification image in multi-class classification, then classification accuracy can be evaluated, but it becomes difficult to evaluate performance stability compared to single-value abnormality detection
Solution Approach 1:
The patent introduces a statistical processing intermediary that aggregates multiple matching degree values into representative statistics (mean, standard deviation, min, max). This intermediary layer transforms the complex multi-class matching degree data into simplified stability metrics, resolving the contradiction between accurate classification evaluation and manageable stability assessment.
2Measurement precision
If only ground truth rate is used to evaluate accuracy, then determination correctness can be measured, but determination speed and performance stability cannot be assessed
Solution Approach 1:
The patent segments the single ground truth rate metric into multiple distinct evaluation dimensions: accuracy (correct determination rate), stability (variation of matching degrees), and speed (determination time). This segmentation allows comprehensive performance assessment while preserving all relevant information without loss.
3Reliability
If matching degrees of multiple classes are considered in multi-class classification, then accurate classification can be achieved, but evaluation of performance stability becomes more complex compared to single abnormality degree calculation
Solution Approach 1:
The patent changes the evaluation parameters from individual class matching degrees to statistical aggregates (mean, standard deviation, minimum, maximum) of matching degrees across all classes. This parameter transformation simplifies the measurement of performance stability while maintaining the reliability benefits of multi-class classification.
Data Source
AI summary
An apparatus includes a calculation unit that calculates a matching degree of each class of multiple classes for each verification image of a plurality of verification images with which a ground truth class is associated by using the machine learning model, a classification unit that classifies the each verification image into any one of the multiple classes based on the matching degree of the each class calculated for the each verification image, and a display processing unit that displays, on a predetermined display device, a separation degree between a statistic of matching degrees of the ground truth class and a statistic of matching degrees of other class that is a class different from the ground truth class, among the matching degrees of the each class calculated for a verification image group associated with a predetermined ground truth class.