Cubic Regularization Optimizer for Nonlinear Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current optimization methods for nonlinear problems, particularly in neural network training, face challenges in finding a reliable and efficient global minimum due to the addition of cubic terms, which forms a non-convex optimization problem.
Innovation Solution
The proposed solution involves performing conjugate gradient iterations with predefined first and second order derivatives, updating residual and H-conjugate vectors, computing eigenvectors, and using these to define a basis matrix for cubic regularization iterations, repeating the process until a stop criterion is met to solve nonlinear optimization problems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a cubic term is added to a regularized quadratic problem to ensure a descent search direction, then the function being optimized is improved, but the optimization problem becomes non-convex and harder to solve globally
Solution Approach 1:
The patent segments the optimization problem into two parts: a regularized quadratic problem (convex) and a cubic term (non-convex). By treating them separately and using the quadratic part as a foundation, the method maintains reliability while managing complexity through structured decomposition of the overall optimization task.
Solution Approach 2:
The patent introduces a regularization parameter that controls the balance between the quadratic and cubic terms. By adjusting this parameter, the method can transition between more convex (reliable) and less convex (more accurate descent) formulations, allowing dynamic control over the trade-off between reliability and problem complexity.
2Measurement precision
If iterative conjugate gradient and cubic regularization methods are used to solve nonlinear optimization problems, then solution accuracy is improved, but computational time and complexity increase
Solution Approach 1:
The patent performs preliminary conjugate gradient iterations to generate an initial search direction and approximate solution before applying the full cubic regularization method. This preliminary action reduces the search space and provides a better starting point, thereby improving final accuracy while reducing the total computational time required.
Solution Approach 2:
The patent uses a limited number of conjugate gradient iterations (partial action) rather than completing them fully, then combines this with cubic regularization. This partial execution of the conjugate gradient method, followed by targeted cubic regularization steps, achieves good solution accuracy without the full computational cost of either method executed to completion.
Data Source
AI summary
A computer determines a solution to a nonlinear optimization problem. A conjugate gradient (CG) iteration is performed with a first order derivative vector and a second order derivative matrix to update a CG residual vector, an H-conjugate vector, and a residual weight vector. A CG solution vector is updated using a previous CG solution vector, the H-conjugate vector, and the residual weight vector. An eigenvector of the second order derivative matrix having a smallest eigenvalue is computed. A basis matrix is defined that includes a cubic regularization (CR) solution vector, a CR residual vector, the CG solution vector, the CG residual vector, and the eigenvector. A CR iteration is performed to update the CR solution vector. The CR residual vector is updated using the first order derivative vector, the second order derivative matrix, and the updated CR solution vector. The process is repeated until a stop criterion is satisfied.