Persistent History Buffer for RNN Training Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training recurrent neural networks (RNNs) is time-consuming and computationally expensive, especially when dealing with large datasets and long sequences of data, due to the need for large recurrent window sizes that are limited by memory and computational constraints.
Innovation Solution
The use of a persistent history buffer that maintains temporal congruity across training batches allows the RNN to access and learn from past data outside the current training window, reducing the need for large window sizes and improving training efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the recurrent window size is increased to improve training accuracy on long sequences, then the training time and computational expense increase significantly
Solution Approach 1:
The patent divides the training process into segments by processing data in batches with smaller recurrent windows, while maintaining a persistent history buffer that stores relevant past information. This segmentation allows the system to achieve long-sequence training accuracy without the computational burden of processing entire long sequences at once.
Solution Approach 2:
The persistent history buffer acts as an intermediary between past data and current processing. It stores and manages historical information that would otherwise require large window sizes, enabling the system to access past context without increasing the computational window size, thus resolving the contradiction between accuracy and training time.
2Reliability
If the recurrent window size is increased to capture long-term dependencies, then the memory requirements increase proportionally
Solution Approach 1:
The patent extracts only the essential historical information needed for long-term dependency recognition and stores it in a persistent history buffer, rather than maintaining entire sequences in memory. This extraction approach reduces memory requirements while preserving the ability to recognize long-term patterns.
Solution Approach 2:
The patent transitions from a temporal dimension approach (increasing window size to capture more time steps) to a buffer storage dimension approach (persistently storing selected historical data). This dimensional change allows the system to access long-term information without proportionally increasing memory usage during active computation.
3Reliability
If the recurrent window size is increased to improve generalization on unseen data, then the computational expense increases significantly
Solution Approach 1:
The system performs preliminary actions by pre-processing and storing relevant historical data in the persistent buffer during training. This preliminary organization of data allows for more efficient computation during inference and testing, improving generalization ability without the continuous computational expense of processing large windows.
Solution Approach 2:
The patent changes the parameter of window size from large to small, while compensating by utilizing the persistent history buffer. This parameter change reduces computational expense while maintaining or improving generalization ability, as the buffer provides the necessary historical context for accurate predictions on unseen data.
Data Source
AI summary
A persistent history buffer may be maintained in training a recurrent neural network such that information from at least one prior group of sequential training parameters within a training sequence is maintained for a subsequent group of training parameters. The persistent history buffer may be provided as an input to the recurrent neural network, and may store a history of a state of the recurrent neural network such as an input, an output and/or the state of a hidden layer. The persistent history buffer may be reset at the end of a sequence of input training parameters, which in a further example may span training input windows and/or batches.