Neural Network Training via Hierarchical Polynomial Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training large, deep neural networks is time-consuming and computationally expensive due to the large-scale, highly non-convex optimization problems involved, with existing techniques offering a disadvantageous trade-off between guarantees of global optimality and scalability.
Innovation Solution
The fast-multidimensional global polynomial solver (FM-GPS) performs semi-definite convex relaxations of constrained polynomial optimization problems, employing algebraic sampling and exploiting structural properties like sparsity and hierarchical structures to efficiently solve nested polynomial optimization problems hierarchically.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing optimization techniques are used for training neural networks, then training can be completed, but the training process is time-consuming and computationally expensive
Solution Approach 1:
The patent segments the complex non-convex optimization problem into multiple convex sub-problems through variable transformation and constraint relaxation. By dividing the original intractable optimization task into smaller, manageable convex components, the system achieves faster computation while maintaining solution quality, directly addressing the training speed vs. training time contradiction.
Solution Approach 2:
The patent transforms the optimization variables and parameters into a different mathematical representation that converts non-convex constraints into convex forms. This parameter transformation enables the use of efficient convex optimization algorithms, significantly reducing computational time and resource requirements while improving training speed.
2Productivity
If existing optimization techniques are used for training neural networks, then training can be completed, but computational costs are high
Solution Approach 1:
The patent replaces the mechanical iterative optimization process with a mathematical transformation approach. By substituting the traditional gradient-based mechanical optimization with an algebraic transformation to convex constraints, the system eliminates the need for repeated iterative computations, thereby reducing computational energy consumption while improving efficiency.
Solution Approach 2:
Through changing the parameter representation and optimization variables, the patent transforms computationally intensive non-convex optimization into efficient convex optimization. This parameter change enables the use of closed-form or near-closed-form solutions, dramatically reducing computational energy requirements.
3Reliability
If existing optimization techniques are used, then training can proceed, but guarantees of global optimality and scalability are compromised
Solution Approach 1:
Instead of directly solving the complex non-convex optimization problem, the patent inverts the approach by transforming the problem into an equivalent convex formulation. This inversion allows the application of well-established convex optimization theory and algorithms, providing reliable global optimality guarantees while managing complexity through mathematical transformation rather than direct computation.
Data Source
AI summary
A processor-implemented method includes approximating an optimization problem for training an artificial neural network as a nested polynomial optimization problem. The method also includes dividing the nested polynomial optimization problem into a sequence of sub-problems. The method further includes hierarchically solving the sequence of sub-problems to train the artificial neural network.