Parallel Hyperparameter Search With Shared Test History
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Hyperparameter tuning for machine learning models is time-consuming and inefficient due to the high-dimensional, nonlinear search space, especially for complex models with many variables, leading to insufficient exploration within a limited timeframe.
Innovation Solution
A computer system executes multiple independent hyperparameter search methods in parallel processes, sharing a memory to store and reuse test results, and includes a global search method to optimize hyperparameter combinations, avoiding duplicates and enhancing exploration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple independenthyperparameter search methods are executed in parallel processes, then the speed and exploration ofhyperparameter combinations is improved, but the complexity of the system increases
Solution Approach 1:
The hyperparameter search process is divided into multiple independent search methods (e.g., grid search, random search, adaptive search) that can be executed in parallel. Each search method operates independently with its own process, allowing simultaneous exploration of different regions of the search space. This segmentation enables the system to test multiplehyperparameter combinations concurrently, significantly improving the overall speed ofhyperparameter tuning without requiring all methods to coordinate complexly.
Solution Approach 2:
The patent introduces a new dimension to the search process by executing multiple search methods in parallel across different processes. Instead of sequentially testinghyperparameter combinations one after another, the system creates a parallel dimension where multiple searches proceed simultaneously. This dimensional change from sequential to parallel execution transforms the time complexity, allowing the system to explore a larger portion of the search space within the same timeframe.
2Loss of time
If duplicate testing ofhyperparameter combinations is avoided by sharing test results in memory, then the efficiency ofhyperparameter tuning is improved, but the requirement for memory sharing and result management increases
Solution Approach 1:
The patent merges the test result storage of multiple independent search methods into a single shared memory structure. Instead of each search method maintaining its own separate results, all processes access a common memory space where test results are stored. This merging eliminates redundant testing by allowing any process to check whether ahyperparameter combination has already been tested before executing the test, thereby avoiding duplicate work and saving time.
Solution Approach 2:
The shared memory acts as a feedback mechanism where test results from any search method are immediately available to other processes. When a process tests ahyperparameter combination, the result is stored in shared memory and can be read by other processes. This feedback loop allows the system to learn from previous tests and avoid redundant executions, improving overall efficiency by preventing duplicate testing across all parallel search methods.
3Measurement precision
If a global search method is added to coordinate and optimizehyperparameter combinations from all search methods, then the quality ofhyperparameter optimization is improved, but the number of processes and coordination complexity increases
Solution Approach 1:
The global search method serves multiple functions simultaneously: it coordinates the search efforts of individual search methods, optimizeshyperparameter combinations by considering results from all processes, and manages the shared memory space. By making the global search method multi-functional, the patent reduces the need for separate dedicated components for each function, thereby improving optimization quality without proportionally increasing system complexity.
Data Source
Figure 1~2
AI summary
According to one aspect, a method implemented by a computer system (SYS) of hyperparameters of a machine learning model is proposed, the computer system (SYS) comprising a processing unit (PU) configured to execute several processes in parallel, the method comprising an execution of several independent hyperparameter search methods in different parallel processes of the processing unit (PU), the results of the tests of the hyperparameter combinations being stored in a memory of the computer system shared between the different processes, and in which each process evaluates whether a searched hyperparameter combination has already been tested by another process from the test results stored in memory, and takes into account, in its own test history, the test results stored in memory if the searched hyperparameter combination has already been tested.