ML Training Time Prediction via Regressor Landmarks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Accurately predicting the training time of machine learning models is challenging due to the instability of training time across different hyperparameter settings and datasets, with variations spanning several orders of magnitude, and existing methods are resource-intensive and lack consistency.
Innovation Solution
The development of meta-learning techniques to train a regressor that predicts training time by exploring the multidimensional hyperparameter space, using meta-features and configuration landmarks to provide a flexible and accurate prediction, allowing for cost-aware hyperparameter tuning and resource allocation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If hyperparameter tuning is performed by repeatedly training models with different hyperparameter values to explore the multidimensional configuration space, then model performance (accuracy, f1 score) is improved, but training time and compute resource consumption increase significantly
Solution Approach 1:
The patent applies preliminary action by pre-training a prediction model on historical training data that includes various hyperparameter configurations and their corresponding training times. This prediction model is built in advance and can quickly estimate training time for new hyperparameter settings without actually training the model, enabling time-aware hyperparameter optimization before actual model training begins.
Solution Approach 2:
The patent uses copying by creating a prediction model that replicates the relationship between hyperparameter configurations and training times based on historical data. Instead of repeatedly training the actual model to explore the hyperparameter space, a copied prediction model is trained once and then used to estimate training times for various hyperparameter settings, significantly reducing the computational burden.
2Measurement precision
If the hyperparameter search space is expanded to explore more configurations for better model performance, then prediction accuracy is improved, but the complexity of the tuning process and resource requirements increase
Solution Approach 1:
The patent applies segmentation by dividing the hyperparameter optimization process into two distinct stages: (1) training a prediction model on historical data that captures the relationship between hyperparameters and training times, and (2) using the trained prediction model to guide the hyperparameter search. This segmentation allows the system to handle large hyperparameter search spaces efficiently by pre-processing the learning patterns from historical data.
3Productivity
If training time prediction is attempted without sufficient historical data, then the tuning process is faster to initiate, but the prediction accuracy decreases and the model may overfit
Solution Approach 1:
The patent applies partial action by using a subset of historical training data that is sufficient to train the prediction model effectively. The system collects training time data from previous model training runs and uses this partial historical information to build the prediction model, balancing the need for sufficient data to avoid overfitting with the desire to keep the initial setup process efficient.
Data Source
AI summary
Herein are techniques for exploring hyperparameters of a machine learning model (MLM) and to train a regressor to predict a time needed to train the MLM based on a hyperparameter configuration and a dataset. In an embodiment that is deployed in production inferencing mode, for each landmark configuration, each containing values for hyperparameters of a MLM, a computer configures the MLM based on the landmark configuration and measures time spent training the MLM on a dataset. An already trained regressor predicts time needed to train the MLM based on a proposed configuration of the MLM, dataset meta-feature values, and training durations and hyperparameter values of landmark configurations of the MLM. When instead in training mode, a regressor in training ingests a training corpus of MLM performance history to learn, by reinforcement, to predict a training time for the MLM for new datasets and/or new hyperparameter configurations.