GRU Inference Engine Using Non-Volatile Memory Arrays
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Artificial neural networks, particularly recurrent neural networks (RNNs), are computationally intensive and require significant data transfer for inferencing operations, which can be resource-intensive and power-consuming, especially when dealing with large datasets and complex models like convolutional neural networks (CNNs) and gated recurrent units (GRUs).
Innovation Solution
Implementing a compute-in-memory architecture using non-volatile memory arrays to perform in-array multiplications within the memory arrays of a non-volatile memory device, specifically for a GRU cell, where inputs are converted into analog voltages and applied to memory arrays to generate the hidden state for the current cycle, reducing the need for external data transfer and enhancing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If traditional von Neumann architecture is used for RNN inference, then computational accuracy is maintained, but data transfer requirements and energy consumption increase significantly
Solution Approach 1:
The patent merges memory and computation functions into a unified in-memory computing architecture. Non-volatile memory arrays perform both weight storage and multiplication operations simultaneously, eliminating the need for separate memory and processing units. This consolidation reduces data transfer between memory and processing units, directly addressing the energy consumption issue while accepting increased architectural complexity.
Solution Approach 2:
The patent introduces analog voltage signals as an intermediary mechanism between digital input data and computational results. Input data is converted to analog voltages that directly interact with memory cell resistances through Ohm's law, enabling multiplication operations without digital processing. This analog intermediary approach reduces energy consumption by avoiding repeated digital-to-analog conversions and data transfers.
2Productivity
If weights are stored in external memory and transferred to processing units, then flexibility and reconfigurability are maintained, but data transfer intensity and processing latency increase
Solution Approach 1:
The patent segments the RNN computation into distinct functional blocks implemented as separate non-volatile memory arrays. Each memory array handles specific weight matrices (e.g., input weights, recurrent weights, output weights), allowing parallel processing of different computational stages. This segmentation enables simultaneous weight updates and inference operations, significantly improving throughput while reducing the time weights need to be transferred from external memory.
Solution Approach 2:
The patent performs weight initialization and updates directly within the non-volatile memory arrays before inference operations begin. Training weights are programmed into the memory cell resistances in advance, preparing the computational engine for rapid inference. This preliminary action eliminates the need to transfer weights from external memory during inference, reducing processing latency and improving overall productivity.
3Loss of energy
If in-memory computing is implemented, then data transfer requirements are reduced, but manufacturing precision and device reliability challenges increase
Solution Approach 1:
The patent changes the physical parameter used to represent weights from digital values to analog resistance values in memory cells. This parameter change enables direct analog multiplication through Ohm's law, reducing energy loss by eliminating digital processing steps. The system compensates for manufacturing precision challenges through training-time adaptation, where weight values are adjusted to account for variations in memory cell characteristics.
Solution Approach 2:
The patent discards the requirement for exact manufacturing precision by using iterative training algorithms that recover and compensate for manufacturing variations. During training, the system adjusts weight values to account for systematic errors in memory cell programming, effectively recovering the desired computational accuracy despite imprecise manufacturing. This approach allows the system to tolerate manufacturing variations while maintaining inference accuracy.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach significantly reduces data transfer requirements and enhances energy efficiency and performance by enabling low-power, high-throughput inferencing operations directly within the memory arrays, making it suitable for resource-constrained devices like IoT and self-driving cars.
Implementation Method 1
The memory arrays perform a plurality of in-array multiplications with the weight values the GRU cell
Data Source
AI summary
A non-volatile memory device includes arrays of non-volatile memory cells that are configured to the store weights for a recurrent neural network (RNN) inference engine with a gated recurrent unit (GRU) cell. A set three non-volatile memory arrays, such as formed of storage class memory, store a corresponding three sets of weights and are used to perform compute-in-memory inferencing. The hidden state of a previous iteration and an external input are applied to the weights of the first and the of second of the arrays, with the output of the first array used to generate an input to the third array, which also receives the external input. The hidden state of the current generation is generated from the outputs of the second and third arrays.