Machine Learning Model Evaluation Using Synthetic Prediction Labels
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The acquisition of labeled data for evaluating the performance of a machine learning model is time-consuming and costly, and generating pseudo-correct data using existing methods also requires significant time and resources.
Innovation Solution
A model evaluation device and method that generate multiple second machine learning models different from a first model subject to evaluation, and evaluate the first model based on prediction labels produced by inputting the same data to both the first model and the second models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If labeled data is acquired through professional research or waiting for predetermined time, then data quality and reliability are improved, but time consumption and cost increase
Solution Approach 1:
The patent uses pseudo-correct data generated by machine learning models as copies of true labeled data. Instead of acquiring authentic labeled data through time-consuming professional research, the system creates synthetic labeled data by having multiple ML models predict labels for unlabeled data, thereby copying the function of true labels without the time and cost overhead.
Solution Approach 2:
The system performs self-labeling by using machine learning models to generate their own prediction labels for unlabeled data. The models serve themselves by creating the labeled data they need for evaluation, eliminating the need for external professional labeling services or waiting for predetermined time periods.
2Loss of time
If pseudo-correct data is generated using existing methods, then time and cost for data acquisition are reduced, but generation time and resource consumption increase
Solution Approach 1:
The patent divides the model evaluation process into separate components: a generation unit that creates pseudo-correct data and an evaluation unit that performs assessment. This segmentation allows parallel processing where multiple ML models can generate predictions simultaneously, improving overall generation efficiency while reducing the burden on any single component.
Solution Approach 2:
The patent combines multiple machine learning models to work together in generating pseudo-correct data. By merging the predictive capabilities of multiple diverse models, the system achieves more reliable label generation while distributing the computational workload, thereby improving both accuracy and generation efficiency.
3Reliability
If multiple diverse machine learning models are generated for evaluation, then evaluation accuracy and reliability are improved, but device complexity increases
Solution Approach 1:
The patent introduces a generation unit as an intermediary component that manages the complexity of multiple ML models. This intermediary orchestrates the diverse models, coordinating their predictions and synthesizing pseudo-correct labels, thereby isolating the complexity from the evaluation process and making the system more manageable while maintaining high evaluation accuracy.
Data Source
AI summary
A model evaluation device 100 of the present disclosure includes a generation unit 121 that generates a plurality of second machine learning models that are different from a first machine learning model subject to performance evaluation, and an evaluation unit 122 that evaluates the first machine learning model on the basis of prediction labels that are output by inputting the same data to the first machine learning model and to each of the second machine learning models. Therefore, the model evaluation device 100 is able to assist decision making by a user.