Synthetic Gradient Model for Neural Network Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training neural networks is inefficient due to the need for forward and backward propagation of gradients, which locks the training of a particular subnetwork to subsequent subnetworks, increasing computational complexity and slowing the training process, especially for networks with many subnetworks.
Innovation Solution
The use of synthetic gradients allows for the training of subnetworks without actual gradient computation, using a synthetic gradient model to approximate the gradient of the objective function, thereby avoiding the need for forward and backward propagation, and enabling independent training of each subnetwork.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If forward and backward propagation of gradients is used for training neural networks, then training accuracy can be achieved, but computational complexity increases and training speed decreases
Solution Approach 1:
The patent divides the neural network into multiple independently train able subnetworks. Each subnetwork can be trained separately using synthetic gradients computed from local information, eliminating the need for global forward and backward propagation. This segmentation reduces computational complexity while maintaining training effectiveness through localized gradient approximation.
2Measurement precision
If forward and backward propagation of gradients is used for training neural networks, then training accuracy can be achieved, but training time increases
Solution Approach 1:
The patent pre-computes synthetic gradients using local information and cached values from previous iterations before actual training updates are needed. This preliminary computation of gradient approximations allows subnetworks to be trained independently and in parallel, significantly reducing overall training time while maintaining acceptable gradient directions for accurate training.
3Measurement precision
If traditional gradient-based training is used, then subnetworks can be trained with accurate gradient information, but the training of a particular subnetwork is locked to subsequent subnetworks
Solution Approach 1:
The patent introduces synthetic gradients as an intermediary mechanism that approximates true gradients using local information and cached values. This intermediary allows each subnetwork to receive gradient-like signals independent of other subnetworks' current state, enabling parallel and independent training while maintaining gradient accuracy through iterative refinement of the synthetic gradient computations.
Data Source
Figure 1
Figure 2A~2B
Figure 3
AI summary
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a neural network including a first subnetwork followed by a second subnetwork on training inputs by optimizing an objective function. In one aspect, a method includes processing a training input using the neural network to generate a training model output, including processing a subnetwork input for the training input using the first subnetwork to generate a subnetwork activation for the training input in accordance with current values of parameters of the first subnetwork, and providing the subnetwork activation as input to the second subnetwork; determining a synthetic gradient of the objective function for the first subnetwork by processing the subnetwork activation using a synthetic gradient model in accordance with current values of parameters of the synthetic gradient model; and updating the current values of the parameters of the first subnetwork using the synthetic gradient.