Model Evaluation Thresholding for Precision-Safe Relearning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques for evaluating trained models are inadequate, leading to potential precision loss during relearning and a lack of flexible evaluation methods.
Innovation Solution
An information processing method that involves obtaining output values from a model, determining a threshold based on reference information, and calculating an index value using a selected data group to evaluate the model's performance, allowing for flexible and targeted evaluation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If relearning is performed without evaluation of the trained models, then the model can be updated, but the precision may become lower than that of the original models
Solution Approach 1:
The patent applies preliminary action by performing model evaluation before relearning. The evaluation unit assesses the trained model's performance using reference data and calculated index values to determine whether the model meets predetermined standards. Only models that pass evaluation proceed to relearning, preventing precision degradation while maintaining efficient update cycles.
2Adaptability or versatility
If flexible evaluation in accordance with the use of the model is performed, then the evaluation accuracy improves, but the evaluation complexity increases
Solution Approach 1:
The patent segments the evaluation process into distinct functional units: an evaluation unit that performs model assessment, a calculation unit that computes index values from input data and model outputs, and a determination unit that compares results against reference information. This segmentation enables flexible, use-specific evaluation while managing complexity through modular design.
Solution Approach 2:
The patent utilizes parameter changes by calculating multiple index values (such as accuracy, precision, recall) based on different combinations of input data and model output data. The system selectively uses reference information and index values according to specific evaluation needs, allowing flexible adaptation to different model usage scenarios without requiring a completely different evaluation system.
Data Source
AI summary
An information processing method includes: an obtaining process of inputting each of plural sets of input data to be subjected to inference processing to a model and obtaining plural output value output by the model and representing inference results respectively corresponding to the plural set of input data, and reference information indicating a reference for evaluation of the model; and a processing process of selecting an evaluated data group to be used in the evaluation of the model, from the plural output values, by using a threshold determined on the basis of the reference indicated by the reference information obtained by the obtaining process and calculating an index value representing an evaluation value of the model by using the evaluated data group selected.