Pipelined Resistive Processing Unit Architecture for Neural Network Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training Deep Neural Networks (DNNs) is computationally intensive and requires massive resources and time due to the intrinsic local and parallel nature of the backpropagation algorithm, hindering their further application.
Innovation Solution
A neural network system utilizing a pipelined architecture with resistive processing units (RPUs) that allows simultaneous feed forward, back propagation, and weight update operations across multiple neurons, optimizing resource utilization and reducing training time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional sequential training method is used, then computational accuracy is maintained, but training time is excessively long and computational resources are wasted due to idle neurons
Solution Approach 1:
The training process is segmented into three distinct phases (feed forward, back propagation, weight update) that can be executed in different cycles. This allows different neurons to be in different phases simultaneously, enabling parallel processing across the network while maintaining the integrity of each training operation.
Solution Approach 2:
The patent introduces a temporal dimension to the training process by implementing pipelined architecture with multiple operation cycles. Instead of processing all neurons sequentially in a single dimension of time, the system distributes neurons across multiple time cycles, allowing simultaneous execution of different training phases across different neuron groups.
2Productivity
If all neurons are engaged simultaneously in different operation modes, then resource utilization is optimized, but control complexity increases
Solution Approach 1:
The system employs periodic action by cycling through different operation modes (feed forward, back propagation, weight update) in a repeating sequence. Each neuron follows this periodic pattern, which simplifies control logic compared to managing arbitrary complex schedules, while still achieving high resource utilization through temporal diversification.
Solution Approach 2:
The control system preliminary assigns specific operation modes to neurons based on their layer and function before the training cycle begins. This preliminary assignment simplifies real-time control by pre-establishing the operational schedule, reducing the complexity of dynamic decision-making during training execution.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach significantly accelerates neural network training by engaging all neurons in each operation cycle, reducing idle resources and shortening training time, while maintaining efficient chip area usage.
Implementation Method 1
each array of weights comprising a plurality of resistive processing units (RPU) having respective settable resistances... an input current from an input of the neuron, and an output voltage pulse at an output of the neuron
Data Source
AI summary
A neural network system comprises a plurality of neurons, comprising a layer of input neurons, one or more layers of hidden neurons, and a layer of output neurons. The system further comprises a plurality of arrays of weights, each array of weights being configured to receive a plurality of discrete data points from a first layer of neurons and to produce a corresponding discrete data point to a second layer of neurons during a feed forward operation, each array of weights comprising a plurality of resistive processing units (RPU) having respective settable resistances. The system includes a neuron control system configured to control an operation mode of each of the plurality of neurons, wherein the operation mode comprises: a feed forward mode, a back propagation mode, and a weight update mode.