Machine Learning Model Training Variability Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models trained on small datasets exhibit high variability in outputs due to sensitivity to training data sequence and division, leading to inconsistent results for the same input data, especially in complex models used for survey data analysis.
Innovation Solution
The method involves training multiple neural networks using different seeds and dataset divisions, calculating accuracy metrics, and ranking independent variables by their correlation strength with the dependent variable to identify robust model parameters, ensuring that models are validated with different data subsets to reduce variability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If repeated holdout sampling is used to train machine learning models on small datasets, then the models can be trained with limited data, but the models exhibit high sensitivity to training data sequence and division, leading to substantially different outputs for the same input data
Solution Approach 1:
The training dataset is divided into multiple distinct training subsets and validation subsets through repeated holdout sampling. Multiple machine learning models are trained on different training subsets and evaluated on corresponding validation subsets. This segmentation allows the system to assess model performance across various data divisions, identifying models that maintain consistent performance regardless of specific data partitioning.
Solution Approach 2:
The system varies multiple parameters including the random seed for data partitioning, the specific division of training and validation sets, and the sequence of training data presentation. By training models with different parameter configurations and selecting those that demonstrate stability across parameter variations, the system achieves reliable model outputs even with limited training data.
Data Source
AI summary
Methods and computer-readable media for repeated holdout validation include collecting independent data representing independent variables; collecting dependent data representing a dependent variable; correlating the independent data with the dependent data; creating a data set comprising the correlated independent and dependent data; generating a plurality of unique seeds; creating a plurality of training sets and a plurality of validation sets; associating each training set with a single validation set; training the neural network a plurality of times with the training sets and seeds to create a plurality of models; calculating accuracy metric values for the models using the validation sets associated with the training sets used to create respective models; performing a statistical analysis of the accuracy metric values; and ranking the independent variables by a strength of correlation of individual independent variables with the dependent variable, when a metric of the statistical analysis exceeds a threshold.