Hyper-parameter adjustment device, hyper-parameter adjustment method, and program

Abstract

The present invention comprises: a first calculation unit that defines a search space as a set of values that can be taken by a group of hyper-parameters related to machine learning or a combinatorial optimization problem, defines a class as a group of problems that can make input / output data formats in the machine learning or combinatorial optimization problem the same, defines instances as individual problems belonging to the class, uses a first instance to generate a partial search space composed of elements of the search space for which a value of a prescribed evaluation function is determined to be good, and subjects the partial search space to the execution of a main component analysis to thereby generate surplus main components of the partial search space; and a second calculation unit that uses the surplus main components to generate an adjusted hyper-parameter that is an optimal value of a group of hyper-parameters for a second instance.

Description

Hyperparameter adjustment device, hyperparameter adjustment method, program

[0001] This invention relates to a technique for determining hyperparameter values ​​related to machine learning and combinatorial optimization problems.

[0002] The accuracy of learning and solving combinatorial optimization problems in machine learning largely depends on hyperparameters. Examples of hyperparameters in machine learning include the learning rate and the hidden layers of a neural network. Examples of hyperparameters in combinatorial optimization problems using Ising machines include the initial temperature and the temperature decay rate.

[0003] One technique for determining hyperparameter values ​​is the technique described in Non-Patent Document 1. The technique in Non-Patent Document 1 efficiently finds the optimal hyperparameter values ​​by using transfer. Transfer will be explained again later using the concepts of class and instance. Here, the hyperparameters that the technique in Non-Patent Document 1 adjusts can be broadly divided into two types. The first is hyperparameters related to the neural network, such as the learning rate and the hidden layers of the neural network (hereinafter referred to as first-kind hyperparameters). The second is hyperparameters derived from the individual problems that the neural network deals with (hereinafter referred to as second-kind hyperparameters).

[0004] The following briefly explains transfer. To do so, the terms "class" and "instance" are defined. A group of problems for which the input / output data formats can be made the same is called a class, and an individual problem belonging to a class is called an instance. In machine learning, for example, "breed classification problems from dog images", "spam email classification problems", and "store sales volume prediction regression problems" correspond to classes. Also, in combinatorial optimization problems, for example, "job shop scheduling problems" and "traveling salesman problems" correspond to classes. Therefore, when solving combinatorial optimization problems using an IJM, the class is a group of problems with the same formulation of the constraint conditions that are the input to the problem, the instance is a problem in which the constraint conditions are specifically specified, and it can be said that an instance is the unit for determining one IJM model. And transfer can be said to be the application of the knowledge obtained when solving the source instance (hereinafter referred to as the first instance) to solving the target instance (hereinafter referred to as the second instance).

[0005] Yang Li, Yu Shen, Huaijun Jiang, Tianyi Bai, Wentao Zhang, Ce Zhang, and Bin Cui, "Transfer Learning based Search Space Design for Hyperparameter Tuning," Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp.967-977, 2022.

[0006] In the technology of Non-Patent Document 1, a promising region, which is a set consisting of combinations of hyperparameter values determined to have good evaluation values when the first instance is solved, is transferred. However, since the evaluation of the second type of hyperparameters is not sufficient, even if one tries to efficiently determine a combination of hyperparameter values using transfer, it may not be possible to obtain a combination of hyperparameter values determined to have good evaluation values for the second instance.

[0007] Therefore, the present invention aims to provide a technique for efficiently determining a more favorable combination of hyperparameter values ​​using transitions.

[0008] One aspect of the present invention includes a first calculation unit that generates a sub-search space consisting of elements of the search space whose value of a predetermined evaluation function is determined to be good, using a first instance, and generates surplus principal components of the sub-search space by performing principal component analysis on the sub-search space; and a second calculation unit that generates adjusted hyperparameters which are the optimal values ​​of the set of hyperparameters for the second instance, using the surplus principal components.

[0009] According to the present invention, it is possible to efficiently determine a more preferable combination of hyperparameter values ​​using transitions.

[0010] This is a block diagram showing the configuration of the hyperparameter adjustment device 100. This is a flowchart showing the operation of the hyperparameter adjustment device 100. This is a block diagram showing the configuration of the second calculation unit 120. This is a flowchart showing the operation of the second calculation unit 120. This is a diagram showing an example of the functional configuration of a computer that realizes each device in the embodiment of the present invention.

[0011] The embodiments of the present invention will be described in detail below. Components having the same function will be numbered identically, and redundant explanations will be omitted.

[0012] <Technical Background> Embodiments of the present invention utilize a transition framework, and principal component analysis is used to realize the transition. Principal component analysis will be explained below.

[0013] Principal component analysis generates a ranking of principal components, which are vectors that best represent the direction of dispersion of a large number of data points. This ranking is based on the magnitude of the eigenvalues ​​of the principal components. The top few principal components in this ranking are designated as representative principal components, and the remaining principal components are designated as redundant principal components. Representative and redundant principal components can be determined, for example, using the cumulative contribution rate. The cumulative contribution rate is obtained by dividing the eigenvalue of a principal component by the sum of the eigenvalues ​​of all principal components, and then adding these values ​​up in descending order of contribution rate. Principal components are arranged in descending order of contribution rate (i.e., eigenvalue), and the contribution rates are added up until the cumulative contribution rate reaches the criterion for distinguishing between representative and redundant principal components. The principal components up to this point are designated as representative principal components. For example, the criterion cumulative contribution rate can be 90%. However, the value of the criterion cumulative contribution rate is not limited to 90%. If the space represented by the representative principal components is considered to represent the direction of data dispersion, then the space represented by the redundant principal components represents the direction orthogonal to that dispersion direction.

[0014] The following describes the algorithm of an embodiment of the present invention for combinatorial optimization problems. In combinatorial optimization problems, the hyperparameters to be adjusted can be broadly divided into two types. The first is hyperparameters related to the Ising machine, and the second is hyperparameters related to the Ising model. Hyperparameters related to the Ising machine are parameters that define the behavior of the Ising machine. On the other hand, hyperparameters related to the Ising model are the weight parameters of the Ising model obtained in the formulation of individual combinatorial optimization problems such as the knapsack problem and the traveling salesman problem. The hyperparameters related to the Ising machine and the hyperparameters related to the Ising model in combinatorial optimization problems correspond to the hyperparameters related to the neural network and the hyperparameters derived from the individual problems handled by the neural network in machine learning, respectively. Therefore, the algorithm of the embodiment of the present invention can also be applied to machine learning.

[0015] The algorithm of the embodiment of the present invention is as follows: <<Algorithm of the embodiment of the present invention>> ------------------------------------------------------------------ Step 1: For the first instance, a solution is found when a certain element of the search space is used, and at the same time, the value of the evaluation function for that element (hereinafter referred to as the evaluation value) is calculated, thereby generating a predetermined number (hereinafter referred to as the first number) of pairs of elements of the search space, solutions, and evaluation values. A sub-search space is generated that contains a predetermined number (hereinafter referred to as the second number) of elements of the search space whose evaluation values ​​are judged to be good from the first number of pairs of elements of the search space, solutions, and evaluation values. Step 2: Principal component analysis is performed on the sub-search space generated in Step 1 to generate the excess principal components of the sub-search space. Step 3: For the second instance, the weights of the excess principal components generated in Step 2 are determined by a predetermined determination method, thereby generating an excess vector which is a linear combination of the excess principal components. Step 4: Candidate hyperparameters are generated using the excess vector generated in Step 3, and the solution is found by solving the second instance using the generated candidate hyperparameters. Step 5: Calculate the value of the objective function for the solution obtained in Step 4. Step 6: If the number of operations in Steps 3 to 5 reaches a predetermined number, the process ends with the candidate hyperparameters corresponding to the best value of the objective function being used as the hyperparameters for the second instance; otherwise, the process returns to Step 3. ------------------------------------------------------------------ In Step 1, for example, the first number can be 16000 and the second number can be 160. That is, the set of hyperparameter values ​​for the first instance in which the evaluation value is in the top 1 percent is used as the subsearch space. In Step 2, for example, with a base cumulative contribution rate of 90 percent, three extra principal components can be generated from the subsearch space consisting of eight hyperparameters.In Step 3, the weights of the extra principal components can be determined using a method that optimizes the parameters using a probabilistic method, such as Optuna. Step 4 can be solved using an Ising machine, such as simulated annealing. The objective function in Step 5 can be, for example, the proportion of constraints in the Ising model that are satisfied by the input solution. In this case, the objective function takes the solution as input and outputs the value obtained by dividing the number of constraints satisfied by the solution by the total number of constraints. In Step 6, the predetermined number of iterations can be set to the number of iterations for which sufficiently good hyperparameters are expected to be obtained, for example, 1000.

[0016] <First Embodiment> The hyperparameter tuning device 100 narrows down the search space using the first instance and adjusts the values ​​of the optimal set of hyperparameters for the second instance. Here, the search space is the set of possible values ​​for a set of hyperparameters relating to a machine learning or combinatorial optimization problem. Furthermore, a class is a group of problems in which the input and output data formats in a machine learning or combinatorial optimization problem can be the same, and an instance is an individual problem belonging to a class.

[0017] The hyperparameter adjustment device 100 will be described below with reference to Figures 1 and 2. Figure 1 is a block diagram showing the configuration of the hyperparameter adjustment device 100. Figure 2 is a flowchart showing the operation of the hyperparameter adjustment device 100. As shown in Figure 1, the hyperparameter adjustment device 100 includes a first calculation unit 110, a second calculation unit 120, and a recording unit 190. The recording unit 190 is a component that appropriately records information necessary for processing by the hyperparameter adjustment device 100.

[0018] The operation of the hyperparameter adjustment device 100 will be explained with reference to Figure 2.

[0019] In S110, the first calculation unit 110 takes the first instance as input and uses the first instance to generate a sub-search space consisting of elements of the search space whose value of a predetermined evaluation function is determined to be good. It then performs principal component analysis on the sub-search space to generate and output the surplus principal components of the sub-search space. The first calculation unit 110 is, for example, a component that executes the processing of Step 1 to Step 2 of the algorithm described in <Technical Background>. The evaluation function is a function that outputs an evaluation value indicating the quality of an element in the search space. For example, it can be constructed using the objective function for the solution result of the first instance when that element is used.

[0020] In S120, the second calculation unit 120 takes the second instance and the surplus principal components output in S110 as input, and uses the surplus principal components to generate and output adjusted hyperparameters, which are the optimal values ​​for the set of hyperparameters for the second instance. The second instance may be a different instance from the first instance. The second calculation unit 120 is, for example, a component that executes the processing of Steps 3 to 6 of the algorithm described in <Technical Background>.

[0021] The second calculation unit 120 will be described below with reference to Figures 3 and 4. Figure 3 is a block diagram showing the configuration of the second calculation unit 120. Figure 4 is a flowchart showing the operation of the second calculation unit 120. As shown in Figure 3, the second calculation unit 120 includes a surplus vector generation unit 121, an objective function calculation unit 122, and a termination condition determination unit 123.

[0022] The operation of the second calculation unit 120 will be explained in accordance with Figure 4.

[0023] In S121, the surplus vector generation unit 121 takes the surplus principal components as input, determines the weights of the surplus principal components using a predetermined determination method, generates a surplus vector which is a linear combination of the surplus principal components with the determined weights as coefficients, and outputs it. The surplus vector generation unit 121 is, for example, a component that executes the processing of Step 3 of the algorithm described in <Technical Background>. Furthermore, a random optimization method can be used as the determination method.

[0024] In S122, the objective function calculation unit 122 takes the surplus vector output in S121 as input, generates candidate adjusted hyperparameters using the surplus vector, solves the second instance using the candidate adjusted hyperparameters, calculates the value of the objective function for the result of solving the second instance, and outputs it. Here, solving the second instance means, for example, calculating the output value of the neural network being trained in machine learning, and in combinatorial optimization problems it means, for example, solving the Ising model using an Ising machine. The objective function calculation unit 122 is, for example, a component that executes the processing of Steps 4 to 5 of the algorithm described in <Technical Background>.

[0025] In S123, the termination condition determination unit 123 determines whether a predetermined termination condition is met. If the termination condition is met, it selects the candidate adjusted hyperparameters corresponding to the best value of the objective function calculated in S122 as the adjusted hyperparameters, outputs the adjusted hyperparameters, and terminates the process. Otherwise, it returns to the process in S121. Here, the termination condition can be, for example, a condition where T is the upper limit of the number of times the process by the second calculation unit 120 is executed, and t is a counter that counts the number of executions, and the counter t is greater than or equal to T. If it can be determined that the value of the objective function has converged, the process may be terminated before the counter t reaches T. The termination condition determination unit 123 is, for example, a component that executes the process in Step 6 of the algorithm described in <Technical Background>.

[0026] According to embodiments of the present invention, it is possible to efficiently determine a more preferable combination of hyperparameter values ​​using transitions. More specifically, embodiments of the present invention enable the rapid search for optimal hyperparameters for a target instance by narrowing down the subsearch space in which optimal hyperparameter values ​​are likely to exist using principal component analysis.

[0027] <Note> The functions realized by the components described herein may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (Central Processing Units), conventional circuits, and / or combinations thereof, programmed to realize the functions described herein. A processor is considered to be circuitry or processing circuitry, including transistors and other circuits. A processor may be a programmed processor that executes a program stored in memory.

[0028] In this specification, circuitry, unit, and means are hardware programmed to perform or execute the functions described herein. Such hardware may be any hardware disclosed herein, or any hardware known to be programmed to perform or execute the functions described herein.

[0029] If the hardware is a processor that is considered to be a type of circuitry, then the circuitry, means, or unit is a combination of hardware and software used to constitute the hardware and / or processor.

[0030] The various processes described above can be carried out by loading a program that executes each step of the above method into the recording unit 2020 of the computer 2000 shown in Figure 5, and then causing the control unit 2010, input unit 2030, output unit 2040, display unit 2050, etc. to operate.

[0031] The program describing this process can be recorded on a computer-readable recording medium. Any computer-readable recording medium can be used, such as a magnetic recording device, optical disc, magneto-optical recording medium, or semiconductor memory.

[0032] Furthermore, this program may be distributed, for example, by selling, transferring, or lending portable recording media such as DVDs or CD-ROMs on which the program is recorded. Alternatively, the program may be stored in the storage device of a server computer and distributed by transferring the program from the server computer to other computers via a network.

[0033] A computer executing such a program may, for example, first store the program recorded on a portable storage medium or a program transferred from a server computer in its own storage device. Then, when processing is to be executed, the computer reads the program stored in its own storage device and executes the processing according to the read program. Alternatively, the computer may directly read the program from the portable storage medium and execute the processing according to that program, or it may sequentially execute the processing according to the received program each time a program is transferred to it from a server computer. Furthermore, the processing may be executed using a so-called ASP (Application Service Provider) type service, where the processing function is realized only by issuing execution instructions and obtaining results, without transferring the program from the server computer to this computer.In addition, the processing may be executed using a so-called SaaS (Software as a Service) type service, where a part of the server computer is made available to the user along with the program. Furthermore, the term "program" in this form includes information used for processing by an electronic computer that is equivalent to a program (data, etc., that is not a direct instruction to the computer but has the property of defining the processing of the computer).

[0034] Furthermore, in this configuration, the device is configured by executing a predetermined program on a computer, but at least a part of these processes may be implemented in hardware.

[0035] The present invention is not limited to the embodiments described above, and can be modified as appropriate without departing from the spirit of the invention. Furthermore, the processes described in the above embodiments may not only be executed in chronological order according to the order described, but may also be executed in parallel or individually as needed, depending on the processing capacity of the device performing the process.

Claims

1. A hyperparameter tuning device comprising:

1. A search space defined as the set of possible values ​​for a set of hyperparameters relating to a machine learning or combinatorial optimization problem; 2. A class defined as a group of problems whose input and output data formats are identical in a machine learning or combinatorial optimization problem; 3. An instance defined as an individual problem belonging to a class; 4. A first calculation unit that generates a sub-search space consisting of elements of the search space that are judged to have good values ​​for a predetermined evaluation function using a first instance; and 5. A second calculation unit that generates adjusted hyperparameters, which are the optimal values ​​for a set of hyperparameters for a second instance, using the surplus principal components.

2. A hyperparameter adjustment device according to claim 1, wherein the second calculation unit includes: an excess vector generation unit that generates an excess vector which is a linear combination of the excess principal components with the weights of the excess principal components determined by a predetermined determination method as coefficients; an objective function calculation unit that solves the second instance using candidate adjusted hyperparameters generated using the excess vector and calculates the value of the objective function for the result of solving the second instance; and a termination condition determination unit that determines whether a predetermined termination condition is met, and if the termination condition is met, sets the candidate adjusted hyperparameter corresponding to the best value of the objective function as the adjusted hyperparameter.

3. A hyperparameter tuning method comprising: a search space defined as the set of values ​​that a set of hyperparameters relating to a machine learning or combinatorial optimization problem can take, a class defined as a group of problems in a machine learning or combinatorial optimization problem whose input and output data formats can be made the same, and an instance defined as an individual problem belonging to a class, the hyperparameter tuning device generating a sub-search space consisting of elements of the search space that are determined to have good values ​​for a predetermined evaluation function using a first instance, and generating surplus principal components of the sub-search space by performing principal component analysis on the sub-search space; and a tuning step in which the hyperparameter tuning device generates tuned hyperparameters that are the optimal values ​​of a set of hyperparameters for a second instance using the surplus principal components.

4. A program for causing a computer to function as a hyperparameter tuning device according to claim 1 or 2.

Images