Stateful Neural Network Circuit With Multi-Scale Memory Registers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current recurrent neural networks are limited in their ability to effectively propagate information across different temporal feature scales, as they utilize a single state variable and can only remember at most a single time point in the past, making them inadequate for real-world applications requiring spatial and temporal pattern recognition.
Innovation Solution
A spatiotemporal residual recurrent network (SRRN) architecture that incorporates state data from multiple previous executions of the neural network, using a register to store and propagate information across different temporal feature scales, allowing the network to remember its history at various time scales and make more sophisticated decisions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single state variable is used in recurrent networks, then the network structure remains simple, but the ability to propagate information across different temporal feature scales deteriorates
Solution Approach 1:
The patent divides the single state variable into multiple state variables, each responsible for capturing information at different temporal feature scales. This segmentation allows the network to maintain separate memory streams for short-term and long-term dependencies, resolving the contradiction by improving information propagation without requiring a completely complex new architecture.
Solution Approach 2:
The patent extends the state representation from one dimension (single state variable) to multiple dimensions (multiple state variables representing different temporal scales). This dimensional expansion enables the network to capture hierarchical temporal patterns while maintaining a structured and manageable network configuration.
2Reliability
If state data from multiple previous executions is stored and used, then pattern recognition capability improves, but memory requirements and computational overhead increase
Solution Approach 1:
The patent implements periodic registers that store state data at different temporal intervals (e.g., every 1st, 2nd, 4th, 8th execution). This periodic sampling strategy allows the network to capture essential temporal patterns while avoiding the need to store and process every single previous state, thus improving pattern recognition without linearly increasing memory requirements.
Solution Approach 2:
The patent uses a selective subset of previous state data rather than all available historical states. By choosing specific temporal intervals and using learnable functions to determine which past states are most relevant, the network achieves effective pattern recognition with a manageable portion of historical information, balancing reliability with resource constraints.
Data Source
AI summary
Some embodiments provide a neural network inference circuit for executing a neural network that includes multiple nodes that use state data from previous executions of the neural network. The neural network inference circuit includes (i) a set of computation circuits configured to execute the nodes of the neural network and (ii) a set of memories configured to implement a set of one or more registers to store, while executing the neural network for a particular input, state data generated during at least two executions of the network for previous inputs. The state data is for use by the set of computation circuits when executing a set of the nodes of the neural network for the particular input.