DNN Training with Dynamic Zero-Reference RPU Crossbars
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training deep neural networks (DNNs) using resistive processing units (RPUs) is computationally intensive and prone to errors due to incorrect estimation and storage of symmetry points, leading to noise and instability in gradient updates, especially when using symmetric RPU devices.
Innovation Solution
The method involves using two tunable RPU crossbar arrays and digital memory arrays to perform gradient updates for DNN training, dynamically estimating reference values on the fly to account for sign changes and noise, reducing the need for a dedicated reference device array and improving accuracy and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a dedicated reference device array is used to store symmetry points, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The patent uses digital memory arrays to store reference values that represent the symmetry points, rather than using dedicated physical reference device arrays. This digital copying approach maintains measurement precision while reducing hardware complexity and resource usage.
Solution Approach 2:
The patent employs tunable RPU crossbar arrays that can dynamically serve multiple functions: storing actual weights, storing reference values, and performing computations. This multi-functionality eliminates the need for separate dedicated reference device arrays, reducing overall device complexity while maintaining precision through software-controlled reference management.
2Manufacturing precision
If symmetric RPU devices are used for DNN training, then manufacturing precision is improved, but adaptability decreases
Solution Approach 1:
The patent introduces asymmetric reference values that are dynamically computed to compensate for the asymmetry in RPU devices. By using asymmetric reference management in software, the system can handle asymmetric physical devices effectively, maintaining manufacturing precision benefits while gaining adaptability to device variations.
Solution Approach 2:
The patent implements dynamic computation of reference values during the training process, allowing the system to adapt to changing conditions and device characteristics. This dynamic approach enables the system to work with both symmetric and asymmetric RPU devices, enhancing versatility while maintaining precision through continuous reference value optimization.
3Reliability
If dynamic reference value computation is implemented, then reliability is improved, but use of energy increases
Solution Approach 1:
The patent computes reference values dynamically but only to the extent necessary for maintaining reliability. By computing references on-demand and using approximate methods when sufficient, the system achieves gradient update stability without excessive energy consumption, balancing reliability improvement with energy efficiency.
Data Source
AI summary
A computer implemented method includes performing a gradient update for a stochastic gradient descent (SGD) of a deep neural network (DNN) using a first set of hidden weights stored in a first matrix comprising a Resistive Processing Unit (RPU) crossbar array. A second matrix comprising a second set of hidden weights is stored in a digital medium. A third matrix comprising a set of reference values is computed upon a transfer cycle of the first set of weights from the first matrix to the second matrix, accounting for a sign-change (chopper). The third matrix is stored in the digital medium. A third set of weights is updated for the DNN from the second matrix when a threshold is reached for the second set of weights, in a fourth matrix comprising a RPU crossbar array.