Higher-Order LSTM Networks with Highway Connections
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
First-order recurrent neural networks, such as LSTM, face limitations in modeling very long-term dependencies in sequential data due to vanishing or exploding gradients during training, which hinders effective sequential modeling in machine learning applications.
Innovation Solution
The implementation of higher-order long short-term memory (LSTM) networks with highway connections in both spatial and temporal domains, allowing for multiple past histories and higher-order feedback paths to assist in training long time dependencies, thereby addressing the gradient issues.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a first-order LSTM network is used, then the network architecture remains simple, but the capability to model very long-term dependencies is limited
Solution Approach 1:
The patent transitions from first-order to higher-order LSTM networks by adding temporal dimensionality. Specifically, it implements second-order LSTM where the hidden state at time t depends not only on the immediate previous state h(t-1) but also on h(t-2), and extends this to nth-order where dependencies reach back n time steps. This dimensional extension in the temporal domain enables modeling of very long-term dependencies while maintaining a systematic architectural framework
2Reliability
If gradient-based learning methods are used to train RNN, then the network can learn sequential patterns, but vanishing or exploding gradients occur preventing effective training
Solution Approach 1:
The patent implements feedback mechanisms through higher-order recurrent connections where past states are fed back into the network at multiple time lags. The nth-order LSTM incorporates feedback from n previous time steps, creating multiple feedback paths that allow gradient information to flow back through different temporal routes. This multi-path feedback structure helps maintain gradient signal strength and prevents vanishing gradients during backpropagation through time
Solution Approach 2:
The patent introduces gate mechanisms as intermediaries that control information flow between different time steps. These gates (including forget gates, input gates, and output gates in the LSTM structure) act as mediators that regulate the transmission of gradients and information through the network. By learning to open or close these gate pathways, the network can preserve gradient signals over long temporal distances while filtering out harmful gradient variations
Data Source
AI summary
A method and apparatus are provided. The method includes configuring a plurality of long short term memory (LSTM) networks, wherein each of the plurality of LSTM networks is at a different network layer, configuring a plurality of memory cells in a spatial domain of the plurality of LSTM networks, configuring the plurality of memory cells in a temporal domain of the plurality of LSTM networks, controlling an output of each of the plurality of LSTM networks based on highway connections to outputs from at least one previous layer and at least one previous time of the plurality of LSTM networks, and controlling the plurality of memory cells based on highway connections to memory cells from the at least one previous time.