Resistive Processing Unit Stochastic Bit Stream Weight Update
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training deep neural networks (DNNs) is computationally intensive and requires significant resources, hindering their further application due to the inefficiencies in existing hardware approaches for accelerating the training process, particularly in implementing weight updates on 2D crossbar arrays of resistive processing units.
Innovation Solution
The use of an array of resistive processing units with AND gates to perform stochastic bit stream operations, simplifying the weight update process by reducing multiplication operations to AND operations and leveraging stochastic translators to manage conductance changes in resistive processing units, allowing for local and parallel updates with O(1) time complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If conventional floating-point training methods are used, then training accuracy is maintained, but training time and computational resources are excessive
Solution Approach 1:
The patent replaces conventional floating-point arithmetic operations with resistive processing units that perform computations through electrical resistance changes. The RPU array substitutes for traditional CPU/GPU arithmetic units, enabling parallel weight updates through physical resistance modulation rather than sequential computational steps, thereby dramatically reducing training time while maintaining accuracy
Solution Approach 2:
The patent changes the fundamental computational parameter from floating-point numbers to resistance values. By representing weights as physical resistance states in the RPU array, the system transforms computational operations into physical processes where weight updates are achieved through controlled resistance changes, enabling simultaneous computation across all weights in parallel
2Speed
If hardware approaches are used to accelerate DNN training, then training speed improves, but hardware complexity increases
Solution Approach 1:
The patent merges multiple functions into the resistive processing unit: weight storage, multiplication, and addition operations are all performed within the same physical device through resistance modulation. This consolidation eliminates the need for separate computational units, reducing hardware complexity while achieving parallel processing speedups
Solution Approach 2:
The resistive processing unit serves multiple purposes simultaneously: it stores weight values, performs multiplication operations through resistance modulation, and executes addition operations through conductance changes. This multi-functionality reduces the overall hardware footprint and complexity compared to dedicated separate units for each function
3Device complexity
If multiplication operations are performed in weight updates, then computational accuracy is maintained, but computational complexity increases
Solution Approach 1:
The patent substitutes multiplication operations with resistive modulation operations. Instead of performing traditional multiplication through arithmetic logic units, the system uses resistance changes to directly encode weight values, eliminating the need for complex multiplication hardware while maintaining computational accuracy through precise resistance control
Data Source
AI summary
An array of resistive processing units (RPUs) comprises a plurality of rows of RPUs and a plurality of columns of RPUs wherein each RPU comprises an AND gate configured to perform an AND operation of a first stochastic bit stream received from a first stochastic translator translating a number encoded from a neuron in a row and a second stochastic bit stream received from a second stochastic translator translating a number encoded from a neuron in a column. A first storage is configured to store a weight value of the RPU, and a second storage is configured to store an amount of change to the weight value of the RPU. When the first stochastic bit stream and the second stochastic bit stream coincide, the amount of change to the weight value of the RPU is added to the weight value of the RPU.