Split Neural Network Weights for Low-Noise In-Memory Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Artificial neural networks are computationally intensive due to extensive data transfers involved in reading and transferring weights between memory and processing units, which can be optimized by implementing compute-in-memory operations using non-volatile memory arrays.
Innovation Solution
Partitioning matrix values into linear combinations of matrices and normalizing small components before programming into memory arrays, followed by rescaling outputs to reduce programming noise and enhance noise tolerance for small weight values.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If weights are stored in non-volatile memory arrays for compute-in-memory operations, then computational efficiency is improved by minimizing data transfers, but programming noise increases and small weight values become difficult to represent accurately
Solution Approach 1:
The patent segments the weight representation by splitting each weight value into two separate components: an integer component and a fractional component. These components are stored in different memory arrays, allowing the system to handle small weight values more effectively. The integer component captures the dominant value while the fractional component preserves precision for small values, thereby resolving the contradiction between computational efficiency and weight value precision.
Solution Approach 2:
The patent changes the parameter representation by transforming weight values from a single continuous value into discrete integer and fractional components. This parameter transformation allows the system to optimize storage and computation for each component type separately, improving overall precision while maintaining compute-in-memory efficiency. The integer component can be stored with higher precision while the fractional component handles the nuanced variations.
2Device complexity
If small weight values are programmed directly into memory arrays, then device complexity is reduced by avoiding additional processing, but programming noise significantly degrades accuracy
Solution Approach 1:
By segmenting small weight values into integer and fractional components, the patent enables separate optimization strategies. The integer component can be programmed with standard precision while the fractional component is handled specially to preserve accuracy. This segmentation maintains relatively simple device architecture while significantly improving small weight accuracy through differential handling of the two components.
Solution Approach 2:
The patent applies local quality by treating the integer and fractional components differently in the programming process. While the overall system remains simple, the fractional component receives enhanced treatment with higher precision programming techniques applied specifically to it. This localized enhancement improves small weight accuracy without significantly increasing overall device complexity.
3Measurement precision
If full precision weights are transferred between memory and processing units, then measurement precision is improved, but data transfer overhead and power consumption increase significantly
Solution Approach 1:
The patent segments weight precision into two parts: integer precision and fractional precision. During compute-in-memory operations, the system can selectively access and transfer only the necessary components based on the specific computational requirements. This segmentation reduces unnecessary data transfer energy while maintaining the required precision for accurate neural network computations.
Solution Approach 2:
By changing the weight parameter from a single high-precision value to separate integer and fractional components, the patent enables differential transfer strategies. The integer component, which contains the dominant value information, can be transferred with lower bandwidth requirements, while the fractional component is accessed locally during computation. This parameter transformation significantly reduces data transfer energy while preserving measurement precision.
Data Source
AI summary
To reduce programming noise for matrix values stored in a memory array for use in an in-array vector-matrix multiplication, such as for a neural network, the matrix is partitioned into a linear combination of matrices that will preserve the output after the combination, with the small value component matrices being normalize to the lager, full range of values before being programmed into the memory arrays. After multiplying each matrix of the combination with the vector by applying a set of bias values, the outputs are rescaled to undo the normalization before adding the individual outputs back together for the final output. This re-scaling causes the effective noise of small weights to be reduced, providing large noise tolerance for these small weight values.