Extended LSTM Networks with Phonological-Semantic Memory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transformer models consume vast amounts of computing power and memory, especially when processing long texts, and are limited to pairwise interactions, making it challenging to capture complex relationships between tokens and abstract concepts.
Innovation Solution
A neural network architecture combining a phonological memory for storing input vectors and a semantic memory for storing semantic information, allowing linear scaling of computations with input length, reducing processor load and memory usage, and enabling the system to process large amounts of text efficiently while maintaining improved performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If Transformer models are used to process long texts, then the model can capture relationships between tokens, but the computing power and memory consumption increase quadratically with sequence length
Solution Approach 1:
The patent segments the attention mechanism into local and global components. Local attention operates on small windows around each token (constant computational cost), while global attention is applied selectively at key positions or after local processing (reduced computational cost). This segmentation allows the model to capture both local and global relationships without quadratic scaling.
Solution Approach 2:
The patent introduces hierarchical processing dimensions: local attention operates at the token level with fixed windows, while global attention operates at the sequence level selectively. This multi-dimensional approach transforms the computational complexity from quadratic O(n²) to linear or near-linear O(n) by adding structural organization to the attention mechanism.
2Measurement precision
If Transformer models use pairwise interactions to compare tokens, then the model can identify patterns, but it cannot capture complex relationships involving more than two tokens or achieve abstraction
Solution Approach 1:
The patent segments the processing into local pattern recognition (pairwise interactions within windows) and global relationship modeling (selective global attention). This allows the model to maintain precise local pattern matching while adding the capability to capture complex multi-token relationships through hierarchical composition of local patterns into global structures.
Solution Approach 2:
The patent adds a hierarchical dimension to token interactions. Instead of only direct pairwise comparisons, the model processes tokens through multiple levels: local window processing captures immediate patterns, while global attention layers compose these into higher-order relationships. This dimensional hierarchy enables abstraction by allowing the model to build complex representations from simpler local patterns.
3Productivity
If the neural network architecture uses linear scaling with input length, then the processing speed and energy efficiency improve, but the model must be designed differently from traditional Transformer architectures
Solution Approach 1:
The patent segments the attention mechanism into manageable local windows and selective global positions. This segmentation enables linear scaling by processing most tokens locally (constant time per token) while applying global attention only where necessary. The architectural complexity is managed through this structured segmentation rather than requiring completely novel designs.
Solution Approach 2:
The patent introduces dynamic selective attention where the model adaptively determines which positions require global attention versus local processing. This dynamic approach optimizes the trade-off between computational efficiency and relationship capture, allowing linear scaling while maintaining the ability to handle complex cases when needed.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Disclosed is an artificial neural network system (200). In one implementation, the neural network system (200) comprises both a phonological memory (102) configured to store input vectors and to retrieve stored input vectors, and a semantic memory (104) configured to store semantic information extracted from input vectors.