Parameter update method, parameter update apparatus, and parameter update program

Abstract

This parameter update method comprises: converting a gradient vector into a local coordinate representation by using an orthonormal basis of a tangent space; updating each of a primary moment vector and a secondary moment vector by using the converted gradient vector; correcting each of the primary moment vector and the secondary moment vector; dividing each of a plurality of elements of the corrected primary moment vector by a square root of each of a plurality of elements of the corrected secondary moment vector; determining an update direction of a parameter by using a result of the division and the orthonormal basis; updating the parameter represented by points of the Riemannian manifold by using the update direction; and outputting the updated parameter.

Description

Parameter update method, parameter update device, and parameter update program.

[0001] This disclosure relates to a technique for updating the parameters of a learning model.

[0002] Adam is widely used as a parameter optimization method in training deep learning models. Non-Patent Document 1 discloses a technique for generalizing Adam to Riemannian manifolds. Specifically, in Non-Patent Document 1, in order to define element-wise division of multiple elements between a first-order moment vector and a second-order moment vector in the tangent space, such division is replaced with division of multiple elements of the first-order moment vector by the norm of the second-order moment vector.

[0003] However, the method described in Non-Patent Document 1 involves division by the norm of the second moment vector instead of the elements of the second moment vector. As a result, information from the second moment vector is lost, making it difficult to efficiently optimize the parameters of the learning model and causing delays in the optimization process.

[0004] Gary Becigneul, Octavian-Eugen, “Ganea, RIEMANNIAN ADAPTIVE OPTIMIZATION METHODS,” Published as a conference paper at ICLR 2019, [Retrieved June 24, 2024], Internet <URL: https: / / arxiv.org / abs / 1810.00760>

[0005] This disclosure aims to provide a technique that suppresses the delay in the time required to optimize the parameters of a learning model when a Riemannian manifold is adopted as a constraint condition for the parameters of the learning model.

[0006] A parameter update method in one aspect of the present disclosure is a parameter update method performed by a computer to optimize the parameters of a learning model, and includes: obtaining a gradient vector for the parameter represented as a point on a Riemannian manifold; converting the gradient vector to a local coordinate representation using an orthonormal basis of the tangent space on the Riemannian manifold; updating a first-order moment vector and a second-order moment vector, respectively, using the converted gradient vector; calculating a corrected first-order moment vector and a corrected second-order moment vector by correcting the first-order moment vector and the second-order moment vector, respectively, using a predetermined decay rate; dividing each of the elements of the corrected first-order moment vector by the square root of the elements of the corrected second-order moment vector; determining the parameter update direction using the result of the division and the orthonormal basis; updating the parameter represented as a point on the Riemannian manifold using the update direction; and outputting the updated parameter.

[0007] This disclosure demonstrates that when a Riemannian manifold is used as a constraint for the parameters of a learning model, the time delay required for parameter optimization can be suppressed.

[0008] This figure shows an example of the configuration of the parameter update device in the embodiment of this disclosure. This flowchart shows an example of the processing of the parameter update device in the embodiment of this disclosure. This figure shows an example of the algorithm in the parameter update device. This figure shows the algorithm of the optimization method described in Non-Patent Document 1. This figure summarizes the experimental results related to the parameter update device in this embodiment.

[0009] (Knowledge forming the basis of this disclosure) To suppress overfitting and improve the efficiency of learning in deep learning, optimization methods have been proposed that impose orthogonal constraints or low-rank constraints on the weight matrix of a neural network. Such optimization methods can be formulated as unconstrained optimization problems on Riemannian manifolds, and gradient descent on Riemannian manifolds can be applied. On the other hand, in current deep learning, an optimization method called Adam is widely used. When Adam is generalized to Riemannian manifolds, there is a problem that element-wise division between first-order moment vectors and second-order moment vectors cannot be defined on the tangent space. Therefore, in Non-Patent Literature 1, such division is replaced by division by the norm of the second-order moment vectors.

[0010] However, the technique described in Non-Patent Document 1 employs division by the norm of the second moment vector, resulting in the loss of information about the two-dimensional moment vector. In particular, this loss of information occurs when a multi-dimensional manifold such as a Stiefel manifold is used as the Riemannian manifold in the technique described in Non-Patent Document 1. Thus, because the information about the two-dimensional moment vector is lost in the method described in Non-Patent Document 1, the optimization of the learning model parameters cannot be performed efficiently, and the optimization process takes a long time.

[0011] This disclosure adopts the following characteristics to address these issues.

[0012] (1) A parameter update method in one aspect of the present disclosure is a parameter update method performed by a computer to optimize the parameters of a learning model, and includes: obtaining a gradient vector for the parameter represented by a point on a Riemannian manifold; converting the gradient vector to a local coordinate representation using an orthonormal basis of the tangent space on the Riemannian manifold; updating a first-order moment vector and a second-order moment vector, respectively, using the converted gradient vector; calculating a corrected first-order moment vector and a corrected second-order moment vector by correcting the first-order moment vector and the second-order moment vector, respectively, using a predetermined decay rate; dividing each of the elements of the corrected first-order moment vector by the square root of the elements of the corrected second-order moment vector; determining the parameter update direction using the result of the division and the orthonormal basis; updating the parameter represented by a point on the Riemannian manifold using the update direction; and outputting the updated parameter.

[0013] In this configuration, multiple elements of the corrected first-order moment vector are divided by multiple square roots, which are the square roots of each element of the corrected second-order moment vector. This corrected second-order moment vector is calculated using a gradient vector converted to local coordinate representation. Therefore, this configuration does not lose information from the second-order moment vector, unlike Non-Patent Document 1. As a result, this configuration can efficiently optimize the parameters of the learning model when a Riemannian manifold is used as a constraint condition for the learning model parameters, and can suppress delays in the time required for optimization. Furthermore, since delays in the time required for parameter optimization are suppressed, this disclosure reduces the processing load on the computer and can improve the computer's functionality.

[0014] (2) In the parameter update method described in (1) above, the Riemannian manifold may be a Stiefel manifold.

[0015] With this configuration, even when a Stiefel manifold is adopted as the Riemannian manifold, the parameters of the learning model can be converged to optimal values ​​in a short amount of time.

[0016] (3) In the parameter update method described in (2) above, the orthonormal basis may be an orthonormal basis defined by a metric derived from the standard inner product of Euclidean space.

[0017] With this configuration, when a Stiefel manifold is adopted as the Riemannian manifold, the gradient vector can be transformed into a local coordinate representation.

[0018] (4) In the parameter update method described in (1) above, the orthonormal basis may be an orthonormal basis defined by the Riemann metric in Riemannian amplicor conditioning.

[0019] This configuration allows us to convert gradient vectors to local coordinate representations when the parameters of the learning model are constrained by low-rank matrices.

[0020] (5) In the parameter update method described in any one of (1) to (4) above, updating and correcting the first moment vector and the second moment vector may be done by updating and correcting in Adam.

[0021] With this configuration, when Adam (Adaptive Moment Estimation) is generalized to a Riemannian manifold, the information of the second moment vector is not lost, and the parameters of the learning model can be converged to the optimal value in a shorter time compared to conventional techniques.

[0022] (6) In the parameter update method described in any one of (1) to (5) above, determining the update direction may include determining the update direction using the result of the division as coordinates and a vector expressed with the orthonormal basis as the basis.

[0023] This configuration allows for precise determination of the update direction.

[0024] This disclosure can also be implemented as a parameter update program that causes a computer to execute each of the characteristic configurations included in such a parameter update method, or as a parameter update system (parameter update device) that operates using this parameter update program. It goes without saying that such a computer program can be distributed via a computer-readable non-temporary recording medium such as a CD-ROM or via a communication network such as the Internet.

[0025] (7) A parameter update device in another aspect of the present disclosure is a parameter update device for optimizing the parameters of a learning model, and includes a processor that performs the following: obtaining a gradient vector for the parameters represented as points on a Riemannian manifold; converting the gradient vector to a local coordinate representation using an orthonormal basis of the tangent space on the Riemannian manifold; updating a first-order moment vector and a second-order moment vector, respectively, using the converted gradient vector; calculating a corrected first-order moment vector and a corrected second-order moment vector by correcting the first-order moment vector and the second-order moment vector, respectively, using a predetermined decay rate; dividing each of the elements of the corrected first-order moment vector by the square root of the elements of the corrected second-order moment vector; determining the parameter update direction using the result of the division and the orthonormal basis; updating the parameters represented as points on the Riemannian manifold using the update direction; and outputting the updated parameters.

[0026] (8) A parameter update program in another aspect of the present disclosure is a parameter update program that causes a computer to function as a parameter update device for optimizing the parameters of a learning model, and causes the computer to perform the following: obtain a gradient vector for the parameter represented by a point on a Riemannian manifold; convert the gradient vector to a local coordinate representation using an orthonormal basis of the tangent space on the Riemannian manifold; update the first moment vector and the second moment vector, respectively, using the converted gradient vector; calculate a corrected first moment vector and a corrected second moment vector by correcting the first moment vector and the second moment vector, respectively, using a predetermined decay rate; divide each of the elements of the corrected first moment vector by the square root of the elements of the corrected second moment vector; determine the parameter update direction using the result of the division and the orthonormal basis; update the parameter represented by a point on the Riemannian manifold using the update direction; and output the updated parameter.

[0027] The embodiments described below are all specific examples of this disclosure. The numerical values, shapes, components, steps, and order of steps shown in the following embodiments are examples only and are not intended to limit this disclosure. Furthermore, among the components in the following embodiments, those not described in the independent claim representing the highest-level concept will be described as optional components. In addition, the contents of each embodiment can be combined.

[0028] (Embodiment) Figure 1 is a diagram showing an example of the configuration of a parameter update device 1 in an embodiment of the present disclosure. The parameter update device 1 is a device that optimizes the parameters of a learning model. The parameter update device 1 is a computer such as a cloud server, edge server, personal computer, tablet computer, or portable computer. The parameter update device 1 includes a processor 10 and a memory 20. The parameter update device 1 may also include a communication interface, a display for displaying processing results, and an operating device for inputting information. The processor 10 is, for example, a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The processor 10 includes an acquisition unit 11, an update unit 12, and an output unit 13. The acquisition unit 11 to the output unit 13 are realized by the processor 10 executing a parameter update program stored in the memory 20.

[0029] The acquisition unit 11 acquires the gradient vector of the parameter represented by a point on the Riemannian manifold. The acquisition unit 11 only needs to acquire the parameter from the parameter storage unit 21. The parameter storage unit 21 stores the parameter being updated. The initial value of the parameter is determined, for example, randomly.

[0030] The learning model is, for example, a neural network. In this case, the parameters are, for example, weight values ​​and bias values. The learning model is not limited to a neural network. The learning model can be any machine learning model whose parameters are optimized by an optimization algorithm. For example, the learning model can be a random forest, a support vector machine, etc.

[0031] The gradient vector is the gradient vector of the objective function used in an optimization algorithm. The objective function is a function that defines the error between the ground truth data and the inference data.

[0032] The update unit 12 updates the gradient vector acquired by the acquisition unit 11 using an optimization algorithm. The optimization algorithm is, for example, Adam (Adaptive Moment Estimation). The update unit 12 adopts a Riemannian manifold as a constraint condition for the parameters.

[0033] The update unit 12 converts the gradient vector into a local coordinate representation using an orthonormal basis of the tangent space in the Riemannian manifold. The first moment vector and the second moment vector are updated using the gradient vector converted into a local coordinate representation. Hereinafter, the gradient vector converted into a local coordinate representation will be called the "local gradient vector". The update unit 12 calculates the corrected first moment vector and the corrected second moment vector by correcting the first moment vector and the second moment vector, respectively, using a predetermined damping coefficient. The update unit 12 calculates the square root of each of the multiple elements of the corrected second moment vector. The update unit 12 divides each of the multiple elements of the corrected first moment vector by the calculated multiple square roots. The update unit 12 determines the parameter update direction using the division result and the orthonormal basis. The update unit 12 updates the parameters represented by points in the Riemannian manifold using the update direction.

[0034] The output unit 13 outputs the parameters updated by the update unit 12. The output unit 13, for example, stores the updated parameters in the parameter storage unit 21.

[0035] The acquisition unit 11, update unit 12, and output unit 13 repeat this process a predetermined number of times to optimize the parameters.

[0036] Figure 2 is a flowchart showing an example of the processing of the parameter update device 1 in an embodiment of the present disclosure. Figure 3 is a diagram showing an example of the algorithm in the parameter update device 1. Hereinafter, the flowchart of Figure 2 will be described with reference to Figure 3 as appropriate. In Figure 2, the processes of steps S3 and S4 and the processes of steps S2, S5, S6, and S7 are executed in parallel. However, both processes may be executed in series. The processing branches in step S2 because of the local gradient vector c calculated in step S2.ｔ This is because it is used in both steps S2 and S5. When both processes are executed serially, the flowchart in FIG. 2 may be executed in accordance with the order of the algorithm in FIG. 3. In this case, lines 3, 4, and 5 in FIG. 3 correspond to steps S1 and S2, line 6 in FIG. 3 corresponds to step S3, line 7 in FIG. 3 corresponds to step S5, line 8 in FIG. 3 corresponds to step S4, line 9 in FIG. 3 corresponds to step S6, line 10 in FIG. 3 corresponds to steps S7, S8, S9, and S10, and lines 2, 11, and 12 in FIG. 3 correspond to steps S11 and S12.

[0037] (Step S1) Acquisition unit 11 acquires a gradient vector. For example, acquisition unit 11 may acquire the parameter being updated from parameter storage unit 21 and obtain the gradient vector by calculating the gradient vector of the point on the Riemannian manifold corresponding to the acquired parameter. The acquisition unit 11 may randomly generate the initial value of the parameter.

[0038] In the algorithm of FIG. 3, x0 is the initial value of the parameter and is a point on the Riemannian manifold. β1 is the exponential decay rate of the first moment vector m. β2 is the exponential decay rate of the second moment vector v. The exponential decay rates β1 and β2 each take a value between 0 and 1. The step size α ｔ is the learning rate. t is an integer from 1 to T. R is a mapping function that maps a point on the Riemannian manifold and the tangent vector at that point to a point on the Riemannian manifold. M represents the Riemannian manifold, and TM represents the tangent bundle.

[0039] g in the fourth line ｔ is the gradient vector. f in the fourth line is the objective function. g in the fourth line ｔ = gradf(x ｔ－１ ) represents the gradient vector of the point x ｔ－１ on the Riemannian manifold. The point x ｔ－１ is the point on the Riemannian manifold indicated by the parameter at time (t - 1).

[0040] (Step S2) Update unit 12 converts the gradient vector g ｔ to a local coordinate representation to obtain a local gradient vector c ｔ ｔThis is obtained. This process is represented by the fifth line of the equation in Figure 3. e １ ｔ , e ２ ｔ , ..., e ｎ ｔ is an orthonormal basis in the tangent space. n is the dimension of the tangent space. The orthonormal basis is represented by the equation in the third row of Figure 3. In the third row, B is the point x of the Riemannian manifold. ｔ－１ For point x ｔ－１ This is a function that gives an orthonormal basis for the tangent space of . <・,・>x ｔ－１ Point x ｔ－１ The tangent space Tx ｔ－１ This is the dot product at M. That is, the update unit 12 is at point x ｔ－１ Input this into function B, and create an orthonormal basis (e １ ｔ , e ２ ｔ , ..., e ｎ ｔ The update unit 12 obtains the gradient vector g. ｔ and an orthonormal basis (e １ ｔ , e ２ ｔ , ..., e ｎ ｔ By taking the inner product with ), the local gradient vector c ｔ To obtain.

[0041] (Step S3) The update unit 12 updates the first moment vector m. This process is represented by the equation in the 6th line of Figure 3. The update unit 12 updates the first moment vector m at time (t-1). ｔ－１ And the local gradient vector c at time (t) ｔ The first-order moment vector m is updated by weighting and averaging these values ​​with an exponential decay rate β1. The initial value of the first-order moment vector m is, for example, the zero vector.

[0042] (Step S4) The update unit 12 corrects the first-order moment vector m to obtain a corrected first-order moment vector. This process is represented by the equation in the 8th line of Figure 3. m^ ｔ is the corrected first-order moment vector at time (t). The subscript t in β1 in the denominator means β1 raised to the power of t.

[0043] (Step S5) The update unit 12 updates the second moment vector v. This process is represented by the equation in the 7th line of Figure 3. The update unit 12 updates the second moment vector v at time (t-1) ｔ－１ and the local gradient vector c at time (t) ｔ By weighting the squares of and with the exponential decay rate β2, the second moment vector v ｔ Update.

[0044] (Step S6) The update unit 12 corrects the second moment vector v to obtain a corrected second moment vector. This process is represented by the equation in the 9th line of Figure 3. v^ ｔ v is the corrected second moment vector at time (t). The update unit 12 is the second moment vector v ｔ (1-β2 ｔ Divide by ) and obtain the corrected second moment vector v^ ｔ To obtain.

[0045] (Step S7) The update unit 12 calculates the corrected second moment vector v^ ｔ Calculate the element-wise square root of the expression. The element-wise square root is √v^ in the 10th row of Figure 3. ｔ It is expressed as follows. For example, since the corrected second moment vector has n elements, the square root of each of the n elements is calculated.

[0046] (Step S8) The update unit 12 calculates the corrected first-order moment vector m^ ｔ and the square root of the corrected second moment vector √v^ ｔ Divide and element by element. Element by element is shown in the 10th row of Figure 3, m^ ｔ / (√v^ ｔ +ε) ｉ It is expressed as follows: ε is a constant to prevent division by zero. i is an index that specifies an element. Here, i = 1, 2, ..., n. In other words, element-wise division is the corrected first moment vector m^ ｔ The i-th element is the square root of the corrected second moment vector √v^ ｔ This involves dividing by the sum of the i-th element and ε.

[0047] (Step S9) The update unit 12 determines the parameter update direction from the division result for each element. The update direction is the Σ in the 10th row of Figure 3. ｉ＝１ ｎ (m^ ｔ / √v^ ｔ +ε) ｉ e ｔ ｉ This is represented by the tangent space Tx. ｔ－１ This is the tangent vector to M. The update unit 12 is the corrected first moment vector m^ ｔ and the square root of the corrected second moment vector √v^ ｔ The update direction is obtained by calculating the sum of the products of the division results of multiple elements with an orthonormal basis and with .

[0048] (Step S10) The update unit 12 updates the parameters by substituting the update direction calculated in step S9 and the current parameters into the mapping function R. The updated parameters are shown in row 10 of Figure 3, x ｔ It is represented by the update unit 12 in the update direction (Σ ｉ＝１ ｎ (m^ ｔ / √v^ ｔ +ε) ｉ e ｔ ｉ ) with step size α ｔ The value obtained by multiplying by and the current point x ｔ－１ By substituting and into the mapping function R, the updated parameter x ｔ Calculate.

[0049] (Step S11) The update unit 12 uses the updated parameter x calculated in step S10. ｔ The parameters are stored in the parameter storage unit 21.

[0050] (Step S12) The update unit 12 determines whether the parameter update process satisfies the termination condition. The termination condition is that the number of parameter updates reaches a predetermined number of updates. If the number of updates reaches the predetermined number of updates, the update unit 12 determines that the termination condition is satisfied (YES in step S12) and terminates the process. The predetermined number of updates is set to a value in which the parameter is expected to converge. On the other hand, if the number of updates is less than the predetermined number of updates, the update unit 12 determines NO in step S12 and returns the process to step S1. In this way, the processes from steps S1 to S12 are repeated to obtain the optimal value of the parameter.

[0051] Figure 4 shows algorithm 400 of the optimization method described in Non-Patent Document 1. Algorithm 400 differs from the algorithm in Figure 3 in the 4th, 5th, 8th, and 9th lines. Algorithm 400 uses the second moment vector v ｔ is the local gradient vector c ｔ Not the square of the gradient vector g ｔ It is calculated using the square of the norm (5th line). This gives the second moment vector v ｔ This becomes the norm, and the corrected second moment vector v^ ｔ It also becomes the norm (line 7). In line 8, the corrected second moment vector v^ ｔ The square root of the corrected first moment vector m^ ｔ The parameter x is divided and updated. ｔ This is calculated. This corrected second moment vector v^ ｔ The square root of is a norm, not a vector. Algorithm 400 defines element-wise division of the first-order moment vector and the second-order moment vector on the tangent space by substituting element-wise division of the second-order moment vector with division by the norm of the second-order moment vector. As a result, algorithm 400 loses information about the second-order moment vector, making parameter optimization inefficient.

[0052] In contrast, the parameter update device 1 calculates the corrected first-order moment vector m^ ｔ Multiple elements of multiple square roots √v^ ｔis divided by, but these multiple square roots √v^ ｔ are the square roots for each of the multiple elements of the corrected second moment vector v^ ｔ . This corrected second moment vector v^ ｔ is calculated using the local gradient vector c ｔ . Therefore, the parameter update device 1 does not lose the information of the second moment vector like the algorithm 400. As a result, when the parameter update device 1 adopts the Riemannian manifold as the constraint condition of the parameters of the learning model, it can efficiently optimize the parameters of the learning model and suppress the delay in the time required for optimization.

[0053] Fig. 5 is a diagram summarizing the experimental results regarding the parameter update device 1 in the present embodiment. In this experiment, the errors in each of the methods of "Stiefel SGD", "Proposed SGD", "Stiefel Adam", and "Proposed SGD" were measured. In Fig. 5, the smaller the numerical value, the better the accuracy. "Proposed SGD" is a method in which the method of the present disclosure is applied to SGD (Stochastic Gradient Descent), and "Proposed Adam" is a method in which the method of the present disclosure is applied to Adam. Hereinafter, both methods are referred to as proposed methods. "Stiefel SGD" and "Stiefel Adam" are algorithms proposed specifically for the Stiefel manifold. "Stiefel SGD" and "Stiefel Adam" are algorithms proposed by Kong.

[0054] The proposed method is an algorithm applicable to a general Riemannian manifold. The proposed method realizes classification accuracy equal to or higher than that of Kong's method. In particular, in CIFAR-100, an accuracy improvement of 1.01 points with "Proposed SGD" and 0.57 points with "Proposed Adam" was achieved.

[0055] The following modification examples can be adopted in the present disclosure.

[0056] (1) The present disclosure may adopt a Stiefel manifold as a Riemannian manifold. In this case, the update unit 12 uses, as an orthonormal basis, an orthonormal basis defined by a metric induced from the standard inner product of the Euclidean space to convert the gradient vector g ｔ into local coordinate representation.

[0057] The tangent space at a point X of the Stiefel manifold S ｔ (n, p) = {X ∈ R ｎ×ｐ | X Ｔ X = Ip} can be expressed as T Ｔ S ｎ－ｐ (n, p) = {XB + X⊥C | B ∈ Skew(p), C ∈ R Ｔ}, where the matrix X⊥ ∈ R ｎ×（ｎ－ｐ） satisfies X ｐ×ｐ ⊥ X⊥ = I Ｔ and X Ｘ X⊥ = O. Here, using the metric g ｔ (U, V) = Tr(U （ｎ－ｐ）×ｐ V) induced from the standard inner product of the Euclidean space as the metric g on the Stiefel manifold, the number 1 can be taken as an orthonormal basis of the tangent space T Ｘ S Ｔ (n, p). Ｘ S ｔ (n, p).

[0058]

[0059] Here, S ｉ，ｊ ｐ，ｐ is a p×p dimensional matrix with the (i, j) component being 1, the (j, i) component being -1, and other components being 0, and E ｉ，ｊ ｎ－ｐ p is an (n - p)×p dimensional matrix with the (i, j) component being 1 and other components being 0. X⊥ can be calculated by applying, for example, QR decomposition using Householder reflections to X.

[0060] (2) The update unit 12 may convert the gradient vector g ｔ into local coordinate representation using an orthonormal basis defined by the Riemannian metric in Riemannian preconditioning.

[0061] In recent years, the increasing size of deep learning models has made it difficult to fine-tune them for subsequent tasks under limited GPU resources. Therefore, a method called Parameter Efficient Fine-Tuning (PEFT), which efficiently adapts large, pre-trained models to subsequent tasks, is being researched.

[0062] Low-Rank Adaptation (LoRA) is one of the most widely used PEFT techniques. In LoRA, the trained weights W0 ∈ R of each layer of the neural network are used. ｄ×ｋ When fine-tuning for a subsequent task, update the weights using W0 + ΔW = W0 + BA Ｔ This can be expressed using low-rank matrix factorization, as shown below. Here, A∈R ｋ×ｒ , B∈R ｄ×ｒ This is a matrix of rank r ≪ min(d, k).

[0063] There is a method (scaled AdamW) for correcting the gradient by applying the Riemann metric g shown in Equation 2 to the low-rank matrix (B, A) of LoRA.

[0064]

[0065] Here, U = (U Ｂ , U Ａ ), V = (V Ｂ , V Ａ ) ∈ R ｄ×ｒ ×R ｋ×ｒ The point (B, A) ∈ R ＊ ｄ×ｒ ×R ＊ ｋ×ｒ It is the tangent vector at R ＊ ｄ×ｒ is the set of fully rank real sequences. This metric was proposed for the low-rank matrix completion problem and is the gradient ∇ of the objective function f in Euclidean space. Ｂ f, ∇ Ａ Correct f as shown in Math 3.

[0066]

[0067] Therefore, the corrected gradient gradf = (grad Ｂf,grad Ａ f) is a vector belonging to the horizontal space H(B, A) (Equation 5), which is determined by the equivalence class shown in Equation 4.

[0068]

[0069]

[0070] Here, GL(r) is the set of r×r-dimensional invertible matrices.

[0071] Considering the metric g, the tangent space R at point (B, A) ｄ×ｒ ×R ｋ×ｒ We can choose number 6 as the orthonormal basis for this.

[0072]

[0073] Here, X Ｂ = B(B Ｔ B) －１／２ , X Ａ = A ( A Ｔ A) －１／２ The orthonormal basis shown in equation 6 is an orthonormal basis defined by the Riemannian metric in Riemannian preconditioning.

[0074] (3) In the above embodiment, the optimization algorithm was Adam, but it may also be SGD.

[0075] (4) A Riemannian manifold may be a Grassmann manifold, a space of positive definite symmetric matrices (SPD manifold), a hyperbolic manifold, a sphere, or a space of double stochastic matrices.

[0076] This disclosure is useful in the field of machine learning technology because it enables efficient parameter optimization.

Claims

1. A parameter update method performed by a computer to optimize the parameters of a learning model, comprising: obtaining a gradient vector for the parameter represented as a point on a Riemannian manifold; converting the gradient vector to a local coordinate representation using an orthonormal basis of the tangent space on the Riemannian manifold; updating a first-order moment vector and a second-order moment vector, respectively, using the converted gradient vector; calculating a corrected first-order moment vector and a corrected second-order moment vector by correcting each of the first-order moment vector and the second-order moment vector using a predetermined decay rate; dividing each of the elements of the corrected first-order moment vector by the square root of the elements of the corrected second-order moment vector; determining the parameter update direction using the result of the division and the orthonormal basis; updating the parameter represented as a point on the Riemannian manifold using the update direction; and outputting the updated parameter.

2. The parameter update method according to claim 1, wherein the Riemannian manifold is a Stiefel manifold.

3. The parameter update method according to claim 2, wherein the orthonormal basis is an orthonormal basis defined by a metric derived from the standard inner product of Euclidean space.

4. The parameter update method according to claim 1, wherein the orthonormal basis is an orthonormal basis defined by the Riemannian metric in Riemannian preconditioning.

5. The parameter update method according to claim 1 or 2, wherein updating and correcting the first moment vector and the second moment vector respectively are updates and corrections in Adam.

6. The parameter update method according to claim 1 or 2, wherein determining the update direction includes determining the update direction by a vector that expresses the divided result as coordinates and the orthonormal basis as specified.

7. A parameter update device for optimizing the parameters of a learning model, comprising a processor that performs the following: obtaining a gradient vector for the parameter represented as a point on a Riemannian manifold; converting the gradient vector to a local coordinate representation using an orthonormal basis of the tangent space on the Riemannian manifold; updating the first moment vector and the second moment vector, respectively, using the converted gradient vector; calculating a corrected first moment vector and a corrected second moment vector by correcting the first moment vector and the second moment vector, respectively, using a predetermined decay rate; dividing each of the elements of the corrected first moment vector by the square root of the elements of the corrected second moment vector; determining the parameter update direction using the division result and the orthonormal basis; updating the parameter represented as a point on the Riemannian manifold using the update direction; and outputting the updated parameter.

8. A parameter update program that causes a computer to function as a parameter update device for optimizing the parameters of a learning model, the program to perform the following: obtain a gradient vector for the parameter represented by a point on a Riemannian manifold; convert the gradient vector to a local coordinate representation using an orthonormal basis of the tangent space on the Riemannian manifold; update the first moment vector and the second moment vector using the converted gradient vector; calculate a corrected first moment vector and a corrected second moment vector by correcting the first moment vector and the second moment vector, respectively, using a predetermined decay rate; divide each of the elements of the corrected first moment vector by the square root of the elements of the corrected second moment vector; determine the update direction of the parameter using the result of the division and the orthonormal basis; update the parameter represented by a point on the Riemannian manifold using the update direction; and output the updated parameter.

Images