Near-Memory MAC Architecture for Input-Stationary RNNs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Recurrent neural networks (RNNs) face challenges in reducing weight reloading and input fetching complexity, leading to data transfer bottlenecks and high energy consumption due to impracticality of storing vast data in processor caches.
Innovation Solution
Implementing a near-memory-compute (NMC) architecture with an input-stationary dataflow in a compute-in-memory (CIM) cell, where input vectors and weights are stored in memory and accessed only once per time step, reducing control complexity and energy consumption by minimizing repeated data transfers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional compute-in-memory architecture is used for RNN, then computing capability is provided, but weight reloading and input fetching complexity increases leading to data transfer bottlenecks
Solution Approach 1:
The patent inverts the conventional weight-stationary dataflow by implementing input-stationary dataflow. Instead of reloading weights for each computation, the input vector remains stationary in the MAC circuit while weights are streamed from memory. This inversion eliminates the complex weight reloading process and reduces data transfer operations.
Solution Approach 2:
The patent performs preliminary action by loading the input vector into the MAC circuit at the beginning of the time step and keeping it stationary throughout the computation. The weights are then streamed from memory and processed against the pre-loaded input vector, eliminating the need for repeated weight reloading and input fetching during computation.
2Speed
If processor caches are used to store data, then data access speed is improved, but storing vast data becomes impractical
Solution Approach 1:
The patent moves the computation from the processor dimension to the memory dimension by implementing compute-in-memory architecture. The MAC circuit is integrated into the memory device, allowing computations to be performed directly where data is stored, eliminating the need for large processor caches and reducing data transfer bottlenecks.
Solution Approach 2:
The patent introduces an intermediary MAC circuit between memory and processor. This intermediary component performs the actual computation in-memory, acting as a mediator that processes data without requiring it to be loaded into processor caches, thus solving the capacity limitation of traditional caches.
3Productivity
If repeated data transfers are performed for RNN operations, then computations can be executed, but energy consumption increases
Solution Approach 1:
The patent ensures continuity of useful action by maintaining the input vector in the MAC circuit throughout the time step and continuously processing it against streamed weights. This eliminates repeated data transfers and keeps the computation pipeline continuous, reducing energy consumption while maintaining productivity.
Solution Approach 2:
The patent implements self-service by having the MAC circuit process computations using data already present in its buffers (input vector and hidden vector) without requiring continuous external data transfers. The system serves itself by utilizing locally available data for computation, reducing energy-consuming data transfer operations.
Data Source
AI summary
A memory device is provided. The memory device comprises a multiply-and-accumulate (MAC) circuit and a post processing circuit. The MAC circuit comprises vector engine circuits that store a first input vector of a current time step of a recurrent neural network (RNN) and a first hidden vector of a previous time step of the RNN. The vector engine circuits perform MAC operations of the first input vector, the first hidden vector and a weight matrix. The post processing circuit generates a second hidden vector of the current time step of the RNN according to results of the MAC operations.