Encoder-Decoder RNN with External Memory Bank for Time-Series Forecasting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine learning models, particularly recurrent neural networks (RNNs), struggle to effectively learn long temporal patterns in time-series data, such as those required for accurate weather forecasting, due to memory limitations and the inability to capture complex spatial cross-correlations between variables.
Innovation Solution
A computer-implemented method using a sequence-to-sequence architecture with an encoder-decoder configuration and a location-addressable memory bank to enhance RNNs, allowing for temporal skip connections and improved memory management, enabling the model to learn and predict long-term patterns and anomalies in multivariate time-series data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Duration of action of moving object
If RNNs with LSTM and GRU cells are used to process time-series data, then the model can learn temporal patterns, but the model fails to learn long temporal patterns spanning over a few hundred instances due to memory limitations
Solution Approach 1:
The patent introduces an external memory bank that adds a new dimension for storing temporal information, separate from the RNN's internal hidden state. This external memory allows the model to access long-term patterns spanning hundreds of instances by mapping memory addresses to temporal sequences, effectively extending the temporal horizon beyond what standard RNN cells can maintain.
Solution Approach 2:
The external memory bank acts as an intermediary between the RNN and long-term temporal patterns. Instead of relying solely on the RNN's internal memory mechanisms (LSTM/GRU cells), the model uses the external memory bank to store and retrieve distant temporal information, mediating the connection between current processing and historical patterns.
2Duration of action of moving object
If the memory bank size is increased to store longer feedback sequences, then the model can capture longer temporal patterns, but the complexity of the model increases
Solution Approach 1:
The patent segments the temporal information processing by separating the RNN's short-term memory function from the external memory bank's long-term storage function. This segmentation allows each component to specialize: the RNN handles immediate temporal dynamics while the external memory bank manages extended historical patterns, reducing the overall complexity burden on any single component.
Solution Approach 2:
The patent extracts the long-term memory storage function from the RNN architecture itself and places it in an external memory bank. This extraction allows the RNN to remain relatively simple while delegating the complex task of maintaining long temporal sequences to the external memory system, which can be independently optimized.
3Adaptability or versatility
If standard RNNs are used for multivariate time-series forecasting, then the model can process sequential data, but the model cannot learn spatial cross-correlations between different variables
Solution Approach 1:
The patent merges the external memory bank with the encoder-decoder RNN architecture to create a unified system that handles both sequential processing and spatial relationships. The external memory stores multivariate temporal patterns while the RNN components process spatial cross-correlations between variables, and their integration enables simultaneous learning of both temporal and spatial dependencies.
Solution Approach 2:
The external memory bank serves multiple functions: it stores long-term temporal patterns, provides feedback to the RNN, and enables the model to learn both temporal dynamics and spatial cross-correlations. This multi-functionality allows a single architectural addition to address multiple limitations of standard RNNs.
Data Source
AI summary
A computer-implemented method forecasts a timeseries. The method includes loading and running a machine learning model. The machine learning model includes an encoder recurrent neural network (RNN) mapping an input sequence into a fixed-dimensionality vector c and a decoder RNN decoding the vector to produce an intermediate sequence. The model includes a fully connected feed-forward layer (FC-FFL) to produce an output sequence. The machine learning model is run concomitantly. Values of a given input sequence are coupled to produce a given output sequence in output of the FC-FFL. Values of a feedback sequence are stored in a location-addressable memory bank. The memory addresses of the memory bank are mapped onto a temporal sequence of the feedback sequence. Values stored are read to retrieve values of the feedback sequence. The retrieved values are fed to the decoder RNN as the model is being run to obtain the given output sequence.