Artificial Neural Network Hidden State Memory Scaling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing artificial neural networks (ANNs) used in sequential data processing, such as natural language, video frames, and time series, face inefficiencies in memory usage and computational time due to their poor scaling in terms of memory and inference time.
Innovation Solution
The proposed solution involves an ANN that processes a sequence of input data tokens with a hidden state memory and a set of weights, both scaling as O(N), where N is the size of the hidden state memory or the number of weights. The ANN performs logical operations on the hidden state memory to generate an updated hidden state, reducing the total number of compute operations to scale below O(N^1.5).
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional ANNs are used for sequential data processing, then they can handle natural language, video frames, and time series, but they suffer from poor memory usage and computational time scaling
Solution Approach 1:
The patent transforms the hidden state representation from a one-dimensional vector to a two-dimensional matrix structure. This dimensional change enables the network to process sequential data while reducing computational complexity from O(N^2) to O(N^1.5) through optimized matrix operations, thereby maintaining versatility while improving productivity
2Quantity of substance
If the hidden state memory size N is increased to improve processing capability, then the model can handle more complex sequential data, but the computational operations scale poorly above O(N^1.5)
Solution Approach 1:
The patent changes the structural parameters of the hidden state from a vector to a matrix, and modifies the computational operations to use optimized matrix multiplications. This parameter change allows the system to increase memory size N while keeping inference time scaling below O(N^1.5), effectively decoupling memory capacity from computational cost
3Measurement precision
If the number of weights is increased to improve model accuracy, then the model can capture more complex patterns, but memory usage and computational time scale poorly
Solution Approach 1:
By organizing weights and hidden states into matrix structures rather than vectors, the patent enables more efficient memory layout and computation. This dimensional reorganization allows the model to maintain high accuracy with complex patterns while reducing the effective computational complexity below O(N^1.5) through optimized matrix operations
Data Source
AI summary
An artificial neural network (ANN) receives a sequence of input data tokens. The ANN has a hidden state memory of size N and a set of weights having a size that scales on or below an order of N. The ANN processes each token in the sequence of input data tokens by performing, based on each token, a logical operation on the hidden state memory to generate an updated hidden state memory. Processing the sequence of input data tokens comprises performing a number of compute operations that scales on or below an order of N{circumflex over ( )}1.5. The ANN obtains, from the updated hidden state memory after processing each token, a final hidden state memory. The ANN generates an inference result based on the final hidden state memory.