Hyperparameter Tuning Using Surrogate Neural Network
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional hyperparameter tuning processes, such as those using genetic algorithms, require significant processing volume and time due to the need to sufficiently train multiple neural networks with different hyperparameter sets.
Innovation Solution
A hyperparameter tuning device that includes a learning processing unit to train a learner using a hyperparameter set of a first neural network, and a hyperparameter tuning unit that tunes the hyperparameter set based on the post-learning performance of multiple first neural networks with corresponding hyperparameter sets, using a trained second neural network to predict post-learning performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional genetic algorithm tuning processing is used to determine optimal hyperparameter sets, then the output accuracy of the neural network is improved, but the processing volume and processing time become enormously large
Solution Approach 1:
The patent applies preliminary action by pre-training a learner neural network before the genetic algorithm tuning process. This pre-trained learner can then quickly evaluate hyperparameter sets without requiring full training of each candidate network, significantly reducing the processing volume and time while maintaining accurate fitness evaluation for the genetic algorithm.
2Measurement precision
If multiple neural networks with different hyperparameter sets are trained for fitness calculation, then the tuning accuracy is improved, but the processing time increases enormously
Solution Approach 1:
The patent uses a learner neural network as a copy or surrogate model that approximates the performance evaluation function. Instead of training multiple full neural networks to evaluate fitness, the system uses this copied learner model to quickly predict performance metrics, maintaining tuning accuracy while dramatically reducing the time required to evaluate multiple hyperparameter candidates.
3Reliability
If sufficient training is performed on multiple neural networks to ensure accurate fitness evaluation, then the reliability of hyperparameter selection is improved, but the processing volume becomes excessively large
Solution Approach 1:
By performing preliminary training of the learner network before the genetic algorithm execution, the system ensures that the fitness evaluation mechanism is reliable and accurate. This pre-prepared learner can then be reused across multiple generations and candidates, maintaining reliable hyperparameter selection while avoiding the need to re-train networks repeatedly, thus reducing overall processing volume.
Data Source
AI summary
A learning processing unit that causes a second neural network (NN) to be trained, with a hyperparameter set of a first NN accepted as input, so as to output post-learning performance that is the performance of a trained first NN to which the hyperparameter set is set. A GA processing unit adjusts the hyperparameter set of the first NN by a genetic algorithm, with the hyperparameter set of the first NN handled as entity, the fitness of said algorithm being configured to be a value that corresponds to the post-learning performance of the first NN to which the hyperparameter set is set. In processing in each generation of the genetic algorithm, the post-learning performance of the first NN corresponding to each hyperparameter is acquired using the second NN.