Prediction Performance Curve for Machine Learning Algorithm Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine learning methods face challenges in accurately comparing prediction performance among different algorithms due to errors caused by randomness in data selection and varying sample sizes, leading to inaccurate estimation of prediction performance curves and weights that do not reflect actual errors.
Innovation Solution
A method that builds multiple learning models using varying sample sizes, calculates measured prediction performances, and uses regression analysis to create a prediction performance curve, accounting for variances and weights to improve accuracy in comparing machine learning algorithms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If training data of larger sample size is used to learn a model, then prediction performance is improved, but time taken to learn the model increases
Solution Approach 1:
The patent applies preliminary action by estimating the prediction performance curve before actually training the model with large datasets. Through nonlinear regression analysis on small sample size results, the system predicts what performance will be achieved at different sample sizes, allowing premature determination of sufficient sample size without wasting time training unnecessarily large models.
Solution Approach 2:
The patent implements dynamics by creating a flexible prediction performance curve that adapts to different algorithms and datasets. The curve is not fixed but dynamically generated through nonlinear regression, allowing the system to optimize sample size selection for each specific machine learning algorithm being evaluated.
2Measurement precision
If different weights are given to observed prediction performances in nonlinear regression analysis, then estimation accuracy is improved, but complexity of the analysis increases
Solution Approach 1:
The patent applies parameter changes by introducing weight parameters into the nonlinear regression analysis. Different weights are assigned to observed prediction performances based on their reliability, allowing the system to emphasize more accurate measurements while reducing the influence of noisy or unreliable data points, thereby improving estimation accuracy.
Solution Approach 2:
The patent implements local quality by assigning different weights to different observed prediction performances based on their specific characteristics and reliability. Not all data points are treated equally; instead, each observation receives a weight appropriate to its quality, with higher weights given to more reliable measurements and lower weights to less reliable ones.
3Adaptability or versatility
If multiple machine learning algorithms are used to improve prediction performance, then algorithm versatility is improved, but difficulty of comparing performance among algorithms increases
Solution Approach 1:
The patent applies the intermediary principle by introducing prediction performance curves as a mediating structure that standardizes performance comparison across different algorithms. Instead of directly comparing raw performance metrics that vary by algorithm, the curves provide a unified framework that visualizes and compares performance trajectories, making cross-algorithm evaluation manageable and systematic.
Solution Approach 2:
The patent implements dimensionality change by moving from comparing single-point performance metrics to comparing entire performance curves that plot performance across different sample sizes. This adds a dimensional aspect to the comparison, allowing evaluation of not just current performance but also performance trajectory and efficiency at different data sizes.
Data Source
AI summary
A processor builds a plurality of learning models using training data of a plurality of first sample sizes according to a first machine learning algorithm, and calculates a plurality of measured prediction performances. The processor calculates a plurality of estimated variances on the basis of relationship information indicating the relationship between expected value and variance with respect to prediction performance and the plurality of measured prediction performances. The processor creates a first prediction performance curve through a regression analysis using the plurality of measured prediction performances and the plurality of estimated variances. The processor calculates a first evaluation value on the basis of the first prediction performance curve and a second sample size. The processor compares the first evaluation value with a second evaluation value calculated based on a second prediction performance curve corresponding to a second machine learning algorithm and the second sample size.