Global Weight Constrainer for Neural Network Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training artificial neural networks is computationally expensive due to the need for large datasets and frequent retraining, with existing methods like statistical gradient descent facing challenges in applying global constraints, leading to convergence issues and increased computational burdens.
Innovation Solution
The use of conditional gradient descent with global constraints that approximate the error function as a sloped plane, allowing for simplified calculations and reduced training time by considering all weights, thereby improving convergence speed and precision while reducing network complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If statistical gradient descent is used to reduce training time, then training speed improves, but applying global constraints becomes difficult or impossible leading to convergence problems
Solution Approach 1:
The patent introduces a conditional gradient descent method as an intermediary approach between statistical gradient descent and traditional gradient descent. This method uses all training data like traditional gradient descent but applies conditional updates that mimic the efficiency of statistical methods, thereby maintaining convergence reliability while improving training speed.
Solution Approach 2:
The patent changes the update rule parameters by introducing conditional logic based on whether the current weight satisfies a global constraint. The update condition is modified from unconditional (statistical gradient descent) to conditional (if constraint satisfied), allowing global constraints to be enforced while maintaining training efficiency.
2Reliability
If traditional gradient descent with global constraints is used, then convergence reliability improves, but training time increases due to computational burden
Solution Approach 1:
The patent extracts the global constraint checking and enforcement as a separate conditional step from the main gradient descent loop. By isolating the constraint verification and conditional update logic, the method maintains convergence reliability through proper constraint enforcement while minimizing computational overhead by only performing full gradient calculations when necessary.
3Measurement precision
If more neurons are used to achieve higher precision, then classification accuracy improves, but network complexity and training computational cost increase
Solution Approach 1:
The patent changes the optimization parameters by enforcing global constraints on weight values, which regularizes the learning process. This constraint-based approach improves classification precision by preventing weight values from becoming excessively large or small, thereby achieving better generalization without requiring additional neurons or increased network complexity.
Data Source
AI summary
An architecture for training the weights of artificial neural networks provides a global constrainer modifying the neuron weights in each iteration not only by the back-propagated error but also by a global constraint constraining these weights based on the value of all weights at that iteration. The ability to accommodate a global constraint is made practical by using a constrained gradient descent which approximates the error gradient deduced in the training as a plane, offsetting the increased complexity of the global constraint.