Parallel Neural Network Hyperparameter Search Using Gradient Integration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing machine learning technologies using neural networks face significant challenges in efficiently searching for optimal hyperparameter values, particularly the learning rate, which prolongs the training time due to the need for repeated iterations with varying hyperparameter settings.
Innovation Solution
The proposed information processing apparatus employs multiple processors to calculate error gradients and update weights in parallel, integrating error gradients and using distinct hyperparameter values to optimize the learning process, thereby reducing the number of iterations required to converge on optimal hyperparameter settings.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If hyperparameter search is performed by repeatedly executing machine learning with different hyperparameter values, then model accuracy is improved, but learning time is significantly extended
Solution Approach 1:
The patent segments the hyperparameter search process by dividing it into multiple parallel machine learning operations executed by different processors. Each processor handles a portion of the hyperparameter combinations simultaneously, breaking down the sequential search into concurrent segments that reduce overall execution time while maintaining comprehensive search coverage
Solution Approach 2:
The patent merges multiple parallel machine learning operations into a unified training process. Different processors execute learning with different hyperparameter values on the same training data simultaneously, and their results are integrated to determine the optimal hyperparameters, combining the computational power of multiple processors to solve the hyperparameter optimization problem more efficiently
2Productivity
If multiple processors are used to perform machine learning in parallel, then learning time is reduced, but device complexity increases
Solution Approach 1:
The patent makes each processor universal by enabling them to execute the same machine learning algorithm with different hyperparameter configurations. All processors perform identical learning operations on the same training data but with varied hyperparameters, allowing the system to leverage multiple processors without requiring specialized hardware for each processing unit
Solution Approach 2:
The patent implements feedback mechanisms where each processor reports its learning results (accuracy metrics) back to the control system. The control system collects these feedbacks, compares the performance across different hyperparameter settings, and determines the optimal hyperparameters based on the aggregated feedback from all processors, enabling coordinated optimization
Data Source
AI summary
Each of a plurality of processors enters, to a model representing a neural network and including a common first weight, first data different from that used by the other processors, calculates an error gradient for the first weight, and integrates the gradients calculated by each processor. Each processor stores the first weight in a memory and updates the weight of the model to a second weight based on a hyperparameter value different from those used by the other processors, the integrated error gradient, and the first weight. Each processor enters common second data to the model, compares the evaluation results acquired by each processor, and selects a common hyperparameter value. Each processor updates the weight of the model to a third weight based on the selected hyperparameter value, the integrated error gradient, and the first weight stored in the memory.