Machine Learning Evaluation via Synthetic Data and Probability Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Evaluating machine learning models is challenging due to the difficulty in obtaining large amounts of correct answer labeled data, which affects the reliability and fairness of comparative evaluations, especially when models are trained with different data sets and the training data is unknown.
Innovation Solution
A system that trains multiple machine learning models with varying capabilities using correct answer labeled data, generates evaluation data with unknown labels similar to the training data, and uses an item response theory-based probability model to estimate the capability of evaluation models by optimizing parameters representing both model capabilities and correct answer labels.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If publicly available training data is used for evaluation, then the evaluation process is simple and fast, but the reliability and fairness of comparative evaluations deteriorates due to unknown training data characteristics
Solution Approach 1:
The patent creates synthetic evaluation data by copying and transforming training data through a data generator. This synthetic data replicates the statistical properties of real training data while maintaining privacy and controlling characteristics, enabling reliable evaluations without needing actual training data access.
Solution Approach 2:
The patent introduces a probability model as an intermediary between the evaluation system and the training data. This model parameterizes the distribution of training data, allowing the system to generate evaluation data that statistically mirrors real training data characteristics without direct access to the actual training data.
2Adaptability or versatility
If multiple machine learning models with different capabilities are trained using the same training data, then comparative evaluation becomes possible, but the difficulty of obtaining correct answer labeled data increases
Solution Approach 1:
The patent generates synthetic evaluation data by copying training data through a data generator that replicates the statistical properties. This allows creation of multiple evaluation samples without requiring additional real labeled data, effectively copying the labeling information through the probability model.
Solution Approach 2:
The patent changes parameters of the probability model to generate evaluation data with different characteristics. By adjusting model parameters, the system can create evaluation datasets that match the training data distribution while providing the needed diversity for multi-model comparison.
3Ease of manufacture
If evaluation data is generated by sampling from training data, then the evaluation data is easy to obtain, but the measurement precision of model capability deteriorates due to data leakage
Solution Approach 1:
The patent introduces a probability model as an intermediary that parameterizes the training data distribution. This mediator allows generation of evaluation data that statistically represents training data without direct sampling from it, preventing data leakage while maintaining statistical properties.
Solution Approach 2:
The patent creates synthetic evaluation data by copying training data characteristics through the probability model rather than directly sampling from training data. This copying approach maintains statistical properties while avoiding the data leakage issues associated with direct sampling.
Data Source
AI summary
A non-transitory computer readable recording medium storing an evaluation program for causing a computer to execute a process includes training first machine learning models by using pieces of correct answer labeled training data, generating pieces of evaluation data of which similarity to the pieces of training data is equal to or less than a predetermined value and a correct answer label is unknown, acquiring prediction results by each of the first machine learning models and a second machine learning model to be evaluated, for the pieces of evaluation data, and outputting a parameter that indicates the capability when a probability model that represents a probability that each of the first machine learning models and the second machine learning model obtains the prediction result is optimized by inputting the prediction results to the probability model.