Neural Network Weight Optimization via Mixed-Integer Linear Programming
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Artificial neural networks face challenges in determining global optimality for weight determination between neurons due to non-convex differentiable unconstrained nonlinear optimization problems, which often result in local minima rather than global solutions.
Innovation Solution
The use of mixed-integer linear programming (MIP) to reformulate neural network models with piecewise linear activation functions, allowing for global optimality solutions by transforming constrained nonlinear optimization problems into mixed-integer linear models over bound box constraints for synaptic weights.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional gradient-based optimization methods are used to determine neural network weights, then the optimization process can be implemented with standard algorithms, but the method converges to local minima rather than global optima due to the non-convex nature of the problem
Solution Approach 1:
The patent transforms the continuous weight optimization problem into a discrete mixed-integer linear programming problem by introducing binary variables that represent weight sign patterns. This parameter transformation changes the problem from non-convex continuous optimization to convex MILP optimization, guaranteeing global optimality while maintaining computational tractability through linear constraints and objective functions.
2Measurement precision
If multiple restarts are performed to escape local minima, then the chance of finding a better solution increases, but the computational time and resources increase significantly
Solution Approach 1:
The MILP formulation inherently performs exhaustive search through its constraint structure and optimality conditions, eliminating the need for external restart mechanisms. The solver automatically explores the solution space systematically using branch-and-bound or similar algorithms, ensuring global optimality is found without requiring multiple independent optimization runs.
3Reliability
If the activation functions are modeled as non-differentiable piecewise linear functions, then the global optimality can be achieved through MILP reformulation, but the activation functions become less smooth and may affect network performance
Solution Approach 1:
The patent introduces auxiliary continuous variables and binary indicator variables as intermediaries to model the piecewise linear activation functions within the MILP framework. These intermediary variables enable the representation of non-differentiable activation behavior while maintaining linearity of the overall model, allowing standard MILP solvers to handle the problem efficiently.
Data Source
AI summary
Computer systems, machine-implemented methods, and stored instructions are provided for minimizing an approximate global error in an artificial neural network that is configured to predict model outputs based at least in part on one or more model inputs. A model manager stores the artificial neural network model. The model manager may then minimize an approximate global error in the artificial neural network model at least in part by causing evaluation of a mixed integer linear program that determines weights between artificial neurons in the artificial neural network model. The mixed integer linear program accounts for piecewise linear activation functions for artificial neurons in the artificial neural network model. The mixed integer linear program comprises a functional expression of a difference between actual data and modeled data, and a set of one or more constraints that reference variables in the functional expression.