Neuron-Centric Local Learning Rate for Non-Volatile Memory Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Artificial neural networks (ANNs) experience network freeze-out due to non-ideal responses from non-volatile memory elements, leading to reduced accuracy and a narrow learning rate range, especially when conductance responses are nonlinear or asymmetric, causing weights to dither and become ineffective during the learning process.
Innovation Solution
Implementing a neuron-centric method where each neuron stores a local learning rate, determining the effective learning rate for each synapse jointly by the connected neurons, identifying and suppressing dithering weights by decreasing their effective learning rate, and increasing local learning rates for consistently updating weights, thereby stabilizing network performance and mitigating freeze-out.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional backpropagation with fixed learning rate is used, then learning process is simple, but network performance degrades due to weight dithering and freeze-out
Solution Approach 1:
The patent implements a local learning rate mechanism where each neuron maintains its own learning rate value that is independently adapted based on local weight update patterns. This allows different parts of the network to have different learning rates, preventing weight dithering in specific regions while maintaining overall network performance. The local learning rate is stored in each neuron and used to scale weight updates for connections involving that neuron.
Solution Approach 2:
The learning rate is transformed from a static global parameter to a dynamic local parameter that adapts during training. Each neuron's learning rate is dynamically adjusted based on the sign changes of weight updates, allowing the system to respond to changing training conditions and prevent freeze-out without requiring complex external control mechanisms.
2Productivity
If learning rate is increased to improve convergence speed, then training efficiency improves, but weight dithering increases causing freeze-out
Solution Approach 1:
The patent changes the learning rate parameter from a fixed global value to a dynamic local value that varies by neuron and adapts during training. This parameter transformation allows the system to use higher effective learning rates where needed while maintaining stability where weight dithering occurs, resolving the trade-off between training efficiency and weight stability.
3Ease of manufacture
If NVM conductance response is nonlinear or asymmetric, then device manufacturing is simplified, but learning rate range becomes narrow reducing performance
Solution Approach 1:
The patent introduces feedback mechanisms where the sign of weight updates is monitored and used to adjust local learning rates. This feedback loop compensates for the effects of nonlinear and asymmetric NVM conductance responses, allowing the network to maintain stable learning across a wider range of learning rates despite device imperfections.
Data Source
AI summary
Artificial neural networks (ANNs) are a distributed computing model in which computation is accomplished using many simple processing units (called neurons) and the data embodied by the connections between neurons (called synapses) and the strength of these connections (called synaptic weights). An attractive implementation of ANNs uses the conductance of non-volatile memory (NVM) elements to code the synaptic weight. In this application, the non-idealities in the response of the NVM (such as nonlinearity, saturation, stochasticity and asymmetry in response to programming pulses) lead to reduced network performance compared to an ideal network implementation. Disclosed is a method that improves performance by implementing a learning rate parameter that is local to each synaptic connection, a method for tuning this local learning rate, and an implementation that does not compromise the ability to train many synaptic weights in parallel during learning.