Hyperparameter Tuning via Data Subsampling and Bayesian Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional approaches to hyperparameter optimization in machine-learning models are computationally expensive and time-consuming, especially when dealing with large training data sets, due to the need for extensive function evaluations, which can be impractical for datasets with millions of values.
Innovation Solution
The method involves subsampling the training data to determine an initial set of hyperparameter values using quasi-random points and a Gaussian Process model, then iteratively refining these values through Bayesian Optimization, focusing on a reduced search space around the maximum of the posterior GP, thereby reducing computational costs and speeding up the optimization process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional Bayesian Optimization with Gaussian Process is used to optimize hyperparameters on large training data sets, then model accuracy can be improved, but computational cost and time required grow exponentially
Solution Approach 1:
The training data set is divided into a smaller subset for initial hyperparameter optimization, and the remaining data is used for final evaluation. This segmentation allows the optimization process to run efficiently on smaller data while still achieving accurate results when applied to the full data set.
Solution Approach 2:
Hyperparameter optimization is performed preliminarily on a subset of training data before the full model training. This preliminary optimization establishes good initial hyperparameter values that can be used for subsequent full-data training, avoiding the need to perform expensive optimization on the entire large data set.
2Measurement precision
If conventional Bayesian Optimization with Gaussian Process is used to optimizehyperparameter values, then model accuracy can be improved, but computational resources required become prohibitively expensive
Solution Approach 1:
The training data set is divided into a smaller subset for initial hyperparameter optimization, and the remaining data is used for final evaluation. This segmentation allows the optimization process to run efficiently on smaller data while still achieving accurate results when applied to the full data set.
Solution Approach 2:
hyperparameter optimization is performed preliminarily on a subset of training data before the full model training. This preliminary optimization establishes good initialhyperparameter values that can be used for subsequent full-data training, avoiding the need to perform expensive optimization on the entire large data set.
3Manufacturing precision
If hyperparameter optimization is performed on the complete training data set, then optimalhyperparameter values can be obtained, but the time needed to train the model grows exponentially
Solution Approach 1:
The training data set is divided into a smaller subset for initialhyperparameter optimization, and the remaining data is used for final evaluation. This segmentation allows the optimization process to run efficiently on smaller data while still achieving accurate results when applied to the full data set.
Solution Approach 2:
hyperparameter optimization is performed preliminarily on a subset of training data before the full model training. This preliminary optimization establishes good initialhyperparameter values that can be used for subsequent full-data training, avoiding the need to perform expensive optimization on the entire large data set.
Data Source
AI summary
Systems and methods determine optimized hyperparameter values for one or more machine-learning models. A sample training data set from a larger corpus of training data is obtained. Initial hyperparameter values are then randomly selected. Using the sample training data set and the randomly chosen hyperparameter values, an initial set of performance metric values are obtained. Maximized hyperparameter values are then determined from the initial set of hyperparameter values based on the corresponding performance metric value. A larger corpus of training data is then evaluated using the maximized hyperparameter values and the corresponding machine-learning model, which yields another corresponding set of performance metric values. The maximized hyperparameter values and their corresponding set of performance metric values are then merged with the prior set of hyperparameter values. The foregoing operations are performed iteratively until it is determined that the hyperparameter values are converging to a particular value.