Hybrid Neural Network Training With Backward-Guided Forward Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computing devices with limited resources, such as edge devices, face challenges in achieving satisfactory convergence during neural network training due to the lack of gradient magnitudes in feed-forward methods, sensitivity to exploding and vanishing gradients, and the need for trial-and-error in choosing noise scales, making it difficult to perform effective model training.
Innovation Solution
Implementing a hybrid forward-backward model training approach, utilizing cross-momentum techniques to guide forward training with backward gradients, allowing for resilient parameter initialization and improved convergence, even on resource-constrained devices like NPUs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If pure forward training methods are used on edge devices, then device resource consumption is reduced, but training convergence quality deteriorates due to lack of gradient magnitudes
Solution Approach 1:
The patent combines forward training and backward training into a hybrid approach. The processor performs backward training to generate gradient information, then uses this gradient information to guide subsequent forward training steps. This merging allows edge devices to benefit from both the resource efficiency of forward training and the convergence quality of backward training, resolving the contradiction between resource consumption and training quality.
2Manufacturing precision
If backward training is performed to improve gradient magnitudes, then training convergence improves, but computational complexity and resource requirements increase
Solution Approach 1:
The patent implements partial backward training rather than full backward training. The processor performs backward training for a limited number of steps or on a subset of data, then switches to forward training for the remaining steps. This partial application of backward training provides sufficient gradient information to improve convergence while avoiding the full computational burden, thus resolving the contradiction between convergence quality and computational complexity.
3Reliability
If noise scales are adjusted to improve training stability, then training reliability improves, but the need for trial-and-error increases complexity
Solution Approach 1:
The patent uses gradient information from backward training as feedback to dynamically adjust training parameters during forward training. The gradient magnitudes and directions obtained from backward training provide real-time feedback that guides the selection of noise scales and other training parameters, eliminating the need for manual trial-and-error tuning. This feedback mechanism improves training stability while reducing the complexity of parameter tuning.
Data Source
AI summary
Systems and techniques are described for model training. In some aspects, a computing device can determine a batch list indicating a sequence of trainings for each step of a plurality of steps for training network parameters of a neural network model, wherein the sequence of trainings comprises at least one of one or more backward trainings or one or more forward trainings. The computing device can train, according to the batch list, the network parameters of the neural network model. In some aspects, a computing device can determine a backward gradient based on performing backward training of network parameters of a neural network model and can determine, based on the backward gradient, a scale for forward training of the network parameters of the neural network model. The computing device can apply the scale to the network parameters for forward training of the network parameters of the neural network model.