Pipelined Direct Feedback Alignment for Deep Neural Network Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural network training is hindered by high power consumption and low learning speed due to the back-propagation algorithm's requirement for large memory capacity and sequential processing, making real-time retraining and on-chip learning challenging, especially in mobile devices.
Innovation Solution
The implementation of a Pipelined Direct Feedback Alignment (PDFA) algorithm with a random number generator to reduce external memory accesses by directly propagating errors and updating weights in parallel, utilizing a heterogeneous core architecture for efficient error propagation, weight gradient update, and feed-forward operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If the back-propagation algorithm is used for DNN training, then the training can be performed with a standard algorithm, but the memory capacity and bandwidth requirements increase significantly
Solution Approach 1:
The patent extracts and eliminates the need to store intermediate feed-forward results and propagated errors in memory by performing computations in a single forward pass. The error propagation and weight updates are computed directly during the forward propagation without requiring separate storage of intermediate values, thus reducing memory capacity requirements.
Solution Approach 2:
The forward propagation process is designed to serve multiple functions simultaneously: it performs the standard feed-forward computation, error propagation, and weight gradient update all in one pass. This multi-functionality eliminates the need for separate storage of intermediate results that would otherwise be required for each distinct operation.
2Stability of the object's composition
If the back-propagation algorithm is used for DNN training, then the training can be performed sequentially, but the learning speed decreases
Solution Approach 1:
The patent enables continuous computation by performing error propagation and weight updates in the same forward pass without interruption or sequential waiting. The useful action of training continues uninterrupted through the entire network in a single pass, eliminating the idle time between forward propagation and backpropagation that characterizes traditional sequential training.
Solution Approach 2:
The error propagation values and weight update gradients are computed in advance during the forward pass itself, before the traditional separate backpropagation phase would begin. This preliminary computation of error terms and gradients during the forward traversal enables faster subsequent weight updates without requiring a separate, time-consuming backpropagation pass.
3Reliability
If the back-propagation algorithm is used for DNN training, then the error propagation can be performed layer by layer, but the power consumption increases
Solution Approach 1:
The patent merges the error propagation computation with the forward propagation computation into a single unified process. By combining these two separate passes into one, the total number of computational operations is reduced, leading to lower power consumption while maintaining the accuracy of error propagation through the network layers.
4Loss of information
If the back-propagation algorithm is used for DNN training, then the intermediate results must be memorized, but the bandwidth requirements increase
Solution Approach 1:
The patent extracts and eliminates the need to store intermediate feed-forward results and propagated errors in memory by performing computations in a single forward pass. The error propagation and weight updates are computed directly during the forward propagation without requiring separate storage of intermediate values, thus reducing memory capacity requirements.
Data Source
AI summary
Disclosed herein are an apparatus and method for training a deep neural network. An apparatus for training a deep neural network including N layers, each having multiple neurons, includes an error propagation processing unit configured to, when an error occurs in an N-th layer in response to initiation of training of the deep neural network, determine an error propagation value for an arbitrary layer based on the error occurring in the N-th layer and directly propagate the error propagation value to the arbitrary layer, a weight gradient update processing unit configured to update a forward weight for the arbitrary layer based on a feed-forward value input to the arbitrary layer and the error propagation value in response to the error propagation value, and a feed-forward processing unit configured to, when update of the forward weight is completed, perform a feed-forward operation in the arbitrary layer using the forward weight.