Extended LSTM Matrix Memory for Linear Long-Sequence Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network models, such as LSTMs and Transformers, face limitations in storage capacity, parallelizability, and computational efficiency, particularly when processing long texts and large datasets, leading to prohibitive memory demands and challenges in capturing complex relationships beyond pairwise interactions.
Innovation Solution
The introduction of an extended LSTM with exponential gating mechanisms and matrix memory structures, such as sLSTM and mLSTM, which allow for revising storage decisions, enhance storage capacities, and achieve linear computation and memory complexity, enabling efficient processing of long sequences.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If Transformer models with parallelizable self-attention are used, then parallelizability is improved, but memory consumption and computational complexity grow quadratic with sequence length
Solution Approach 1:
The patent changes the computational parameters from quadratic attention mechanisms to linear recurrent operations. By transforming the self-attention computation into a recurrent state update process, the model achieves linear time and space complexity while maintaining parallelizability through vectorized operations on the recurrent states.
Solution Approach 2:
The patent substitutes the mechanical attention mechanism (computing pairwise interactions between all token pairs) with a recurrent neural network mechanism that processes tokens sequentially through a hidden state. This substitution replaces the quadratic computational structure with a linear recurrent structure that is both memory-efficient and parallelizable.
2Quantity of substance
If LSTM networks are used, then storage capacity is improved through memory cells, but inability to revise storage decisions and limited storage capacities persist
Solution Approach 1:
The patent introduces dynamic gating mechanisms that allow the model to adaptively revise storage decisions at each time step. The forget gate and input gate dynamically control what information to retain or update in the memory cell, enabling the model to flexibly revise storage decisions based on current input and previous state, thus overcoming the static nature of traditional LSTM.
Solution Approach 2:
The patent extends the traditional scalar LSTM memory cell to a matrix-valued memory state. By transitioning from scalar to matrix dimensions, the model gains increased storage capacity and the ability to represent more complex relationships while maintaining the reversible storage capabilities through matrix operations.
3Productivity
If LSTM networks are used, then sequential processing capability is improved, but lack of parallelizability due to memory mixing persists
Solution Approach 1:
The patent segments the sequential processing into independent vector operations that can be parallelized. By representing the hidden state and memory cell as vectors and matrices, the recurrent updates can be computed in parallel across different elements, enabling GPU acceleration while maintaining sequential temporal dependencies.
4Extent of automation
If Transformer models are used, then parallelizable self-attention is achieved, but inability to capture complex relationships beyond pairwise interactions persists
Solution Approach 1:
The patent transitions from pairwise interactions in Transformer attention to higher-order interactions through matrix-valued hidden states. The matrix operations enable the model to capture complex relationships involving multiple tokens simultaneously by performing matrix multiplications and transformations that inherently model higher-order dependencies.
Data Source
AI summary
Disclosed is a long short-term memory (LSTM) enhanced with exponential gating with appropriate normalization and stabilization techniques. Also disclosed are LSTM variants with modified memory structures: (i) sLSTM (104) with a scalar memory, a scalar update, and new memory mixing, and (ii) mLSTM (102) with a matrix memory and a covariance update rule, which is fully parallelizable.