Neural Network Training With Hybrid Search for Global Minima
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network training methods struggle to find global minima efficiently in models with few parameters, particularly in non-convex landscapes, leading to suboptimal performance in applications like advanced driver assistance systems (ADAS).
Innovation Solution
A hybrid training procedure combining gradient descent and gradient-free optimization to identify local minima and select a globally optimal minimum, using momentum-based optimizers like Adam and gradient-free methods to navigate the loss landscape effectively.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional gradient descent methods are used to train neural networks, then training convergence can be achieved, but the method fails to find global minima in non-convex loss landscapes, resulting in suboptimal model performance
Solution Approach 1:
The patent applies periodic action by implementing cyclic learning rate changes and multiple restart techniques. The learning rate is cyclically adjusted to help traverse saddle points and escape local minima, while the model is restarted multiple times from different initializations to explore different regions of the loss landscape, ultimately selecting the solution with the lowest loss value.
Solution Approach 2:
The patent changes key training parameters including learning rate (cyclically adjusted), momentum (reset during restarts), and initialization (multiple random initializations). These parameter changes enable the optimization process to escape local minima and explore the non-convex loss landscape more effectively to find global minima.
2Speed
If the learning rate is cyclically changed to traverse saddle points faster, then convergence speed is improved, but the method loses effectiveness in models with few parameters compared to typical Deep Neural Networks
Solution Approach 1:
The patent performs preliminary action by conducting multiple restarts with different initializations before final model selection. This preliminary exploration of the loss landscape from multiple starting points ensures that the cyclic learning rate method is applied effectively across various regions, compensating for the limited parameter space in small models.
Solution Approach 2:
The patent implements feedback by monitoring the loss value across multiple restarts and cyclic iterations, then selecting the model state with the lowest loss. This feedback mechanism ensures that the cyclic learning rate approach is applied adaptively, maintaining effectiveness even in models with fewer parameters by selecting the best outcome from multiple attempts.
3Productivity
If repetitive restart techniques are used to remove ripples in the loss function, then accelerated convergence is achieved, but the method is restricted to convex function properties and cannot guarantee improved model performance
Solution Approach 1:
The patent segments the training process into multiple independent restart phases, each handling a portion of the exploration of the loss landscape. This segmentation allows the method to apply convex optimization techniques within each restart while the overall multi-restart framework handles the non-convex nature of the complete training process.
Solution Approach 2:
The patent creates a universal training framework that combines multiple techniques (cyclic learning rate, momentum adjustment, multiple restarts, and model selection) into a single multi-functional approach. This unified method can handle both convex and non-convex loss landscapes, making it versatile across different model types and loss function characteristics.
Data Source
Figure 1~2
Figure 3
AI summary
The invention relates to a computer-implemented method (100) for training a neural network, is provided, comprising the following steps. In a first step (200), in a plurality of successive cycles, a plurality of local minima in a loss landscape of a loss function is determined, and in a subsequent step (300), a globally optimal minimum in the loss landscape of the loss function is selected out of the plurality detected local minimum.