Mixed-Precision Memristive Neural Network Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training of artificial neural networks (ANNs) is computationally intensive and requires substantial resources due to the need for high-precision floating-point implementations, which are not efficiently addressed by existing analog training methods using memristive arrays, resulting in reduced training accuracy.
Innovation Solution
A method utilizing mixed-precision training with memristive arrays for forward and backpropagation steps and a digital processing unit for weight updates, allowing digital weight-correction calculations to improve accuracy while retaining speed and efficiency advantages, by applying input signals to row and column lines and using programming signals to update weights.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If high-precision floating-point digital implementation is used for ANN training, then training accuracy is improved, but computational complexity and resource requirements increase significantly
Solution Approach 1:
The patent segments the training process into distinct phases: forward propagation and backpropagation are performed using low-precision analog memristive arrays, while weight updates are performed using high-precision digital processing. This segmentation allows each phase to use the most appropriate precision level, reducing overall computational complexity while maintaining training accuracy.
Solution Approach 2:
The patent applies different precision levels to different parts of the training process. Low-precision analog computation is used for forward propagation and backpropagation where approximate values suffice, while high-precision digital computation is used for weight updates where accuracy is critical. This local quality approach optimizes the balance between accuracy and complexity.
2Device complexity
If low-precision analog training methods using memristive arrays are used, then computational complexity is reduced, but training accuracy deteriorates
Solution Approach 1:
The patent introduces a digital processing unit as an intermediary between the analog memristive arrays and the training algorithm. This intermediary performs weight updates with high precision based on error signals from the analog computation, compensating for the low-precision nature of the analog arrays and maintaining overall training accuracy.
Solution Approach 2:
The patent creates a hybrid system combining analog memristive arrays and digital processing units. This composite approach leverages the speed and parallelism of analog computation for forward/backward propagation while using digital precision for weight updates, achieving both reduced complexity and maintained accuracy.
3Measurement precision
If iterative training with repeated weight updates is performed, then training accuracy is improved, but training time increases
Solution Approach 1:
The patent replaces the traditional digital sequential weight update mechanism with an analog parallel update mechanism using memristive arrays. The analog arrays can update multiple weights simultaneously through parallel electrical operations, dramatically reducing the time required for each training iteration while maintaining accuracy through the hybrid digital-analog approach.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach achieves training accuracy comparable to high-precision floating-point implementations while reducing computational complexity and resource requirements, enabling fast and efficient ANN training.
Implementation Method 1
Each memristive device stores a weight W for a synapse interconnecting a respective pair of neurons in successive neuron layers
Implementation Method 2
These systems perform the forward propagation, backpropagation, and weight-update computations on the memristive arrays by applying signals to the row and/or column lines
Data Source
AI summary
Methods and apparatus are provided for training an artificial neural network having a succession of layers of neurons interposed with layers of synapses. A set of crossbar arrays of memristive devices, connected between row and column lines, implements the layers of synapses. Each memristive device stores a weight for a synapse interconnecting a respective pair of neurons in successive neuron layers. The training method includes performing forward propagation, backpropagation and weight-update operations of an iterative training scheme by applying input signals, associated with respective neurons, to row or column lines of the set of arrays to obtain output signals on the other of the row or column lines, and storing digital signal values corresponding to the input and output signals. The weight-update operation is performed by calculating digital weight-correction values for respective memristive devices, and applying programming signals to those devices to update the stored weights.