Neural Turing Machine Memory in NVM Arrays for Lower-Latency RNNs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Recurrent neural networks, particularly Neural Turing Machines (NTMs), face inefficiencies in processing information due to reliance on external volatile memory resources, which can lead to high power consumption and latency, and lack of computational capabilities in the memory component.
Innovation Solution
Implementing a non-volatile memory (NVM) array to store the NTM matrix, incorporating on-chip computational components, and utilizing threshold-based methods to determine whether to read or write data, thereby optimizing NTM operations and reducing unnecessary memory access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If external volatile memory resources are used to store NTM matrix, then data can be accessed quickly, but power consumption increases and latency occurs
Solution Approach 1:
The patent combines memory storage and computational processing into a single integrated system by implementing NTM operations directly within the non-volatile memory array. This merging eliminates the need for separate volatile memory resources and reduces data transfer between memory and processing units, thereby lowering power consumption while maintaining fast access speeds.
Solution Approach 2:
The non-volatile memory array is equipped with on-chip computational components that enable the memory to perform NTM operations autonomously without requiring external processing units. This self-service capability allows the memory to compute weight values and perform matrix operations internally, reducing the energy overhead associated with data movement and external processing.
2Speed
If external volatile memory resources are used to store NTM matrix, then data access is fast, but latency increases
Solution Approach 1:
By merging memory storage and computational processing into the same physical location within the non-volatile memory array, the patent eliminates the latency associated with data transfer between external volatile memory and processing units. The NTM operations are performed in-place, significantly reducing access latency while maintaining fast data retrieval speeds.
Solution Approach 2:
The patent implements threshold-based pre-filtering of weight values before performing full NTM operations. By comparing weight values against thresholds and skipping operations for values below the threshold, the system performs preliminary filtering that reduces the number of computational steps required, thereby reducing overall latency.
3Measurement precision
If all NTM operations are performed by retrieving data from NVM array, then computational accuracy is maintained, but power consumption and latency increase
Solution Approach 1:
The patent applies partial action by selectively performing NTM operations only on weight values that exceed a predetermined threshold. Instead of processing all weight values uniformly, the system performs computations only when necessary (when weights exceed the threshold), thereby reducing power consumption while maintaining computational accuracy for significant weight values.
Solution Approach 2:
The patent dynamically adjusts the threshold parameter based on the distribution of weight values and computational requirements. By changing the threshold parameter adaptively, the system optimizes the balance between computational accuracy and power consumption, ensuring that only meaningful operations are performed while maintaining the precision required for accurate NTM computations.
4Use of energy by moving object
If threshold-based method is used to skip reading from NVM array, then power consumption reduces, but device complexity increases
Solution Approach 1:
The threshold-based filtering mechanism is implemented as on-chip circuitry within the non-volatile memory array, enabling the memory to autonomously filter weight values and determine which operations require execution. This self-service capability reduces the burden on external controllers, thereby limiting the increase in overall device complexity while achieving significant power savings.
Data Source
AI summary
Recurrent Neural Networks (RNNs) wherein a non-volatile memory (NVM) array provides a memory bank for the RNN. The RNN may be a Neural Turning Machine (NTM) and the memory bank may be an NTM matrix stored in the NVM array. In some examples, an NTM controller sets the size of the NTM matrix based on a storage access granularity of the NVM array. For instance, if the NVM reads and writes data in flash memory unit (FMUs), the NTM controller sets the size of the NTM matrix to correspond to the size of an integer number of FMUs. In some examples, the NVM array includes on-chip NTM circuitry configured to perform at least some NTM read head and write head operations. Threshold-based processing is described that can reduce an amount of NTM data read from the NVM array. In other examples, volatile memory is employed rather than an NVM array.