Analog Neural Network Normalization for MAC Precision
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analog memory-based artificial neural networks face performance issues due to incorrect weight mapping and low input resolution, leading to inaccurate neural network outputs, as existing technologies struggle to achieve precise multiply-and-accumulate operations.
Innovation Solution
Applying a scale factor to input values and the inverse of this scale factor to synaptic weight values stored in non-volatile memory devices, allowing for improved precision of MAC operations without altering the nominal MAC results, thus enhancing prediction accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If weights are encoded with high precision, then neural network output accuracy is improved, but hardware cannot reach such values leading to implementation errors
Solution Approach 1:
The patent applies normalization to transform weight values and input resolutions to optimal ranges that match hardware capabilities. By changing the parameters of weight encoding (scaling to appropriate conductance ranges) and input voltage levels, the system achieves high effective precision without requiring hardware to physically represent extreme precision values, thus resolving the contradiction between desired precision and hardware reachability.
2Measurement precision
If input resolution is increased, then MAC operation precision is improved, but hardware cannot achieve the required resolution
Solution Approach 1:
The patent normalizes input values to optimal voltage ranges that maximize the effective resolution of analog-to-digital converters and match the dynamic range of the crossbar array. By transforming input parameters to appropriate scales, the system achieves high effective resolution without requiring hardware to physically support extreme precision input levels.
3Reliability
If weight mapping is corrected, then neural network performance is improved, but complex normalization operations are required
Solution Approach 1:
The patent performs normalization of weights and inputs as preliminary operations before the main MAC computations. By pre-scaling weights to appropriate conductance ranges and pre-normalizing input vectors to optimal voltage levels, the system ensures correct weight mapping and high neural network performance while keeping the main inference operations simple and hardware-efficient.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This method improves the overall precision of MAC operations and prediction accuracy in analog memory-based artificial neural networks by optimizing weight and input resolution, making the neural network outputs more reliable and hardware-friendly.
Implementation Method 1
analog memory crossbar arrays storing synaptic weights as conductance
Implementation Method 2
voltage provided as inputs to such analog memory crossbar arrays storing synaptic weights as conductance can generate current, which can represent a product or multiplication between the input vector and the synaptic weight matrix
Data Source
AI summary
A scale factor can be determined and applied to input values of an analog neural network implemented by a crossbar array of non-volatile memory devices. An inverse of the scale factor can be applied to synaptic weight values stored by the crossbar array of non-volatile memory devices. Operations by the crossbar array can be performed using the scaled input values and the synaptic weight values. The scale factor can be determined for each row of the crossbar array. The scale factor can be determined using a combination of the input values and the synaptic weight values.