Resistive Processing Unit Array for CNN Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network training methods, particularly for convolutional neural networks (CNNs), are computationally intensive and inefficient, requiring significant resources and time due to the need for extensive modifications of adjustable parameters during the training phase, which hampers both speed and power efficiency.
Innovation Solution
The implementation of a resistive processing unit (RPU) array that performs local data storage and processing, allowing for parallel updates of weights within the RPU devices, eliminating the need for separate storage and processing elements, and enabling accelerated training through forward, backward, and update pass computations using voltage pulses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional neural network training methods are used with separate storage and processing elements, then the system can perform computations, but the training speed is slow and power consumption is high due to extensive data movement and sequential processing
Solution Approach 1:
The patent merges storage and processing functions into a single resistive processing unit (RPU) that combines memory and compute capabilities. This integration eliminates the need for separate storage and processing elements, enabling in-place weight updates and eliminating data movement between storage and processing units, thereby significantly improving training speed while reducing power consumption.
Solution Approach 2:
The RPU performs self-contained computations and weight updates without requiring external processing elements. The resistive crosspoint device itself executes the computational operations and stores weights locally, enabling autonomous processing that accelerates training while minimizing energy consumption by eliminating the need for continuous data transfer between separate components.
2Productivity
If separate storage and processing elements are used in neural networks, then the architecture is modular and flexible, but the computational intensity increases and training becomes less efficient
Solution Approach 1:
The patent combines storage and processing into a single unified RPU structure, simplifying the overall system architecture by eliminating the need for separate memory and processing components. This integration reduces the number of interconnections and data movement pathways, thereby reducing architectural complexity while simultaneously improving training efficiency through in-place operations.
3Adaptability or versatility
If extensive modifications of adjustable parameters are performed during training, then the neural network can learn from data, but the computational burden and time required increase significantly
Solution Approach 1:
The patent performs weight updates in-place within the resistive crosspoint devices during the training process, eliminating the need for separate read-modify-write operations. This preliminary action of updating weights directly in the storage location accelerates the learning process by reducing the time required for parameter modifications, enabling faster convergence while maintaining full learning capability.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach significantly enhances the training speed and efficiency of CNNs by reducing the computational burden and power consumption, enabling broader applications and improved performance in machine learning tasks.
Implementation Method 1
trainable resistive crosspoint devices... resistive processing units (RPUs)... two-terminal RPU... local data storage and local data processing
Data Source
AI summary
Technical solutions are described for implementing a convolutional neural network (CNN) using resistive processing unit (RPU) array. An example method includes configuring an RPU array corresponding to a convolution layer in the CNN based on convolution kernels of the layer. The method further includes performing forward pass computations via the RPU array by transmitting voltage pulses corresponding to input data to the RPU array, and storing values corresponding to output currents from the RPU arrays as output maps. The method further includes performing backward pass computations via the RPU array by transmitting voltage pulses corresponding to error of the output maps, and storing the output currents from the RPU arrays as backward error maps. The method further includes performing update pass computations via the RPU array by transmitting voltage pulses corresponding to the input data of the convolution layer and the error of the output maps to the RPU array.