Online Recurrent Neural Network Training via Gradient Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Recurrent neural networks face challenges in online training due to system-locking issues with deep unrolled networks and limitations in existing gradient-based training methods like BPTT, making them unsuitable for online learning scenarios.
Innovation Solution
A computer-implemented method for training neural networks that separates spatial and temporal gradient components, allowing for independent computation and updating at each time instance, facilitating efficient online training and implementation on hardware accelerators like memristive arrays.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If backpropagation through time (BPTT) is used for training recurrent neural networks, then training accuracy can be improved, but system-locking problems occur and online learning becomes unusable
Solution Approach 1:
The patent segments the gradient computation into two independent components: spatial gradient components (computed layer-by-layer from output to input) and temporal gradient components (computed at each time step independently). This segmentation eliminates the system-locking problem of traditional BPTT while maintaining training accuracy, enabling online learning by allowing incremental updates without requiring the entire sequence to be processed first.
2Quantity of substance
If the network unrolls in time with increasing input sequence length, then more training data is processed, but the network depth becomes very deep causing computational intractability
Solution Approach 1:
The patent segments temporal gradient computation into independent per-time-step operations that can be accumulated online, avoiding the need to unroll the entire network through time. This allows processing of long sequences without creating proportionally deep computational graphs, as gradients are computed and accumulated incrementally at each time step rather than requiring full unrolling.
Solution Approach 2:
The patent computes temporal gradient components at each time step in advance and stores them as eligibility traces, which are then combined with spatial gradient components during the forward pass or at designated update points. This preliminary computation of temporal gradients eliminates the need to maintain deep computational graphs for the entire sequence duration.
3Productivity
If spatial and temporal gradient components are computed independently, then computational time is reduced through parallel processing, but gradient accuracy may be compromised
Solution Approach 1:
The patent segments the gradient into spatial components (affecting weight matrices) and temporal components (affecting hidden states), computing them independently in parallel. The spatial gradients are computed via backpropagation through the network layers, while temporal gradients are computed from the unrolled temporal dynamics. These segmented gradients are then combined to update parameters, achieving both parallel computational speed and accurate gradient estimation.
Data Source
AI summary
The invention is notably directed to a computer-implemented method for training parameters of a recurrent neural network. The network comprises one or more layers of neuronal units. Each neuronal unit has an internal state, which may also be denoted as unit state. The method comprises providing training data comprising an input signal and an expected output signal to the recurrent neural network. The method further comprises computing, for each neuronal unit, a spatial gradient component and computing, for each neuronal unit, a temporal gradient component. The method further comprises updating the temporal and the spatial gradient component for each neuronal unit at each time instance of the input signal. The computing of the spatial and the gradient component may be performed independently from each other. The invention further concerns a neural network and a related computer program product.