Hysteretic Resistive Processing Unit for Neural Network Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural network training is computationally intensive and resource-heavy, with existing technologies facing challenges in optimizing training speed and efficiency due to asymmetry in resistive processing unit (RPU) devices, which affects the balance of up and down conductance changes, leading to inefficiencies in neural network training.
Innovation Solution
Introducing hysteresis into RPU devices to mitigate asymmetry by engineering a delay in conductance state changes, allowing for balanced up and down conductance updates, thereby improving the tolerance to imbalance and enhancing training efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If hysteresis is introduced into RPU devices to mitigate asymmetry, then training efficiency and tolerance to imbalance are improved, but device complexity increases
Solution Approach 1:
The patent applies parameter changes by introducing hysteresis into the RPU device conductance update mechanism. Specifically, the conductance change Δg is modified to include a hysteresis term: Δg = ηξgΔw + (1-ξ)g_hystΔw, where the hysteresis component prevents immediate reversal of conductance changes. This parameter modification balances the asymmetric up and down conductance changes, improving training efficiency while maintaining the basic RPU device structure.
2Reliability
If hysteresis updates are implemented to balance conductance changes, then tolerance to asymmetry increases, but the response time of conductance state changes increases due to delay
Solution Approach 1:
The patent implements partial hysteresis action through the mixing parameter ξ, which controls the proportion of hysteresis effect applied. The conductance update equation Δg = ηξgΔw + (1-ξ)g_hystΔw allows selective application of hysteresis, where ξ=1 gives full hysteresis effect and ξ=0 gives no hysteresis. This partial action approach provides tolerance to asymmetry while controlling the delay in conductance state changes, optimizing the trade-off between reliability and speed.
3Measurement precision
If hysteresis is used to correct imbalance in conductance changes, then training accuracy improves, but the number of update pulses required increases
Solution Approach 1:
The patent implements feedback through the hysteresis mechanism that monitors the current conductance state g and adjusts the conductance change accordingly. The hysteresis term g_hystΔw provides feedback about previous conductance changes, preventing oscillations and ensuring more accurate weight updates. This feedback mechanism improves training accuracy by correcting imbalance in conductance changes, although it may require additional update pulses to achieve convergence.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
The implementation of hysteresis in RPU devices significantly increases the tolerance to asymmetry, allowing for faster and more efficient deep neural network training by stabilizing conductance changes, resulting in accelerated training times and improved accuracy.
Implementation Method 1
A plurality of two-terminal RPUs are hysteretic such that the plurality of two-terminal RPUs each have a conductance state defined by hysteresis
Data Source
AI summary
A technique relates a resistive processing unit (RPU) array. A set of conductive column wires are configured to form cross-points at intersections between the set of conductive row wires and a set of conductive column wires. Two-terminal RPUs are hysteretic such that the two-terminal RPUs each have a conductance state defined by hysteresis, where a two-terminal RPU of the two-terminal RPUs is located at each of the cross-points.