Neural Network Weight Training for Memristive Devices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The transfer of digitally-trained artificial neural network (ANN) weights to memristive devices results in a drastic reduction in accuracy due to the stochastic nature of conductance fluctuations and device variability, leading to imprecision in storing weights as programmed conductance states.
Innovation Solution
A method is introduced where a probability distribution of conductance errors for memristive devices is derived and noise is applied to the training process based on this distribution, incorporating random errors to reflect the specific conductance error distribution of the devices, thereby enhancing the accuracy of ANN training for implementation with memristive device arrays.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If digitally-trained ANN weights are transferred to memristive devices, then the network can be implemented with low power consumption and high speed inference, but the accuracy drastically reduces due to conductance fluctuations and device variability
Solution Approach 1:
The patent applies preliminary action by incorporating conductance error noise into the training process before the weights are transferred to memristive devices. The training method simulates the stochastic conductance fluctuations that will occur during inference, allowing the network to learn robust weight values that account for device variability. This preliminary exposure to noise ensures that when the weights are programmed into memristive devices, the network maintains high accuracy despite conductance variations.
2Measurement precision
If full digital precision weights are used during training, then the network achieves high accuracy on digital hardware, but the accuracy drops significantly when weights are programmed into memristive devices with limited resolution
Solution Approach 1:
The patent applies parameter changes by modifying the training process to include conductance error noise that models the limited resolution and variability of memristive devices. Instead of training with ideal digital precision, the training parameters are adjusted to simulate the stochastic nature of conductance programming. This allows the network to learn weight values that are robust to the manufacturing precision limitations of memristive devices, bridging the gap between digital training precision and analog programming precision.
3Productivity
If standard training methods are used without considering device variability, then the training process is simple and fast, but the trained network performs poorly on memristive hardware
Solution Approach 1:
The patent applies feedback by incorporating a model of conductance error distribution into the training loop. The training process uses feedback from the simulated conductance variations to adjust weight updates, ensuring that the network learns to compensate for device variability. This feedback mechanism allows the training to account for memristive device characteristics without requiring complex hardware-in-the-loop training, maintaining training efficiency while improving inference reliability on actual memristive devices.
Data Source
AI summary
Methods are provided for training weights of an artificial neural network to be implemented by inference computing apparatus in which the trained weights are stored as programmed conductance states of respective predetermined memristive devices. Such a method includes deriving for the memristive devices a probability distribution indicating distribution of conductance errors for the devices in the programmed conductance states. The method further comprises, in a digital computing apparatus: training the weights via an iterative training process in which the weights are repeatedly updated in response to processing by the network of training data which is propagated over the network via the weights; and applying noise dependent on said probability distribution to weights used in the iterative training process.