Hebbian Weight Masking for Efficient Neural Network Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network training methods lead to overly complex models due to over-parameterization, overfitting, and catastrophic forgetting, which increase chip area and power consumption.
Innovation Solution
Implement Hebbian matrices and gradient masks to selectively update synaptic weights based on co-activation thresholds, using binary or ternary masks to reduce unnecessary weight updates and prune unnecessary synapses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional backpropagation training is used to train neural networks, then the network can learn from training data, but the model becomes overly complex with too many parameters leading to overparameterization
Solution Approach 1:
The patent applies local quality by differentiating the treatment of different synapses based on their co-activation patterns. Synapses with high co-activation (indicating important feature detection) are retained and updated, while synapses with low co-activation are pruned. This local differentiation allows the network to maintain training accuracy while reducing overall model complexity by removing unnecessary parameters.
Solution Approach 2:
The patent changes the parameter update mechanism by introducing Hebbian masking that modifies which parameters are updated based on co-activation thresholds. Instead of uniformly updating all weights, the system dynamically adjusts parameter updates based on real-time co-activation measurements, transforming the training process to favor important parameters while discarding redundant ones.
2Reliability
If neural networks are trained with more parameters to improve accuracy, then the network can capture more complex patterns, but the chip area and power consumption increase
Solution Approach 1:
The patent extracts and removes unnecessary parameters from the neural network through Hebbian pruning. By identifying and eliminating synapses with low co-activation, the system reduces the total number of parameters while maintaining the essential computational capacity needed for pattern recognition, thereby reducing power consumption without sacrificing accuracy.
Solution Approach 2:
The system discards redundant parameters that do not contribute to accurate pattern recognition and recovers computational efficiency. The Hebbian masking mechanism allows the network to discard unnecessary weights and neurons, reducing the computational burden and associated power consumption while preserving the functional integrity of the network.
3Adaptability or versatility
If neural networks continuously learn new information, then the network adapts to new data, but it forgets previously learned information due to catastrophic forgetting
Solution Approach 1:
The patent applies preliminary action by establishing Hebbian masks before training new information. The co-activation patterns from previous learning are captured in advance and used to create masking patterns that protect important synapses from being overwritten during new training, preventing catastrophic forgetting while enabling continuous adaptation.
Solution Approach 2:
The system uses feedback from co-activation patterns to guide parameter updates. By continuously monitoring which synapses are actively co-active during new learning and using this feedback to adjust the Hebbian masks, the network protects important existing knowledge while incorporating new information, thus maintaining both adaptability and memory retention.
4Reliability
If all synapses are updated during training, then the network learns from training data, but unnecessary weight updates increase computational complexity
Solution Approach 1:
The patent applies partial action by updating only a subset of synapses rather than all synapses. The Hebbian masking mechanism identifies and updates only those synapses with high co-activation patterns, leaving others unchanged. This partial updating approach maintains learning effectiveness for important features while reducing unnecessary computational operations, thereby improving training efficiency.
Data Source
Figure 1~2
Figure 3~4
Figure 5
AI summary
The present disclosure relates to a method of training, by a computing system (400), an artificial neural network (402) comprising a plurality of layers of neurons, the neurons of each layer being coupled to one or more neurons of one or more adjacent layers via synapses, the method comprising: forward propagating one or more first training data points through the network (402) in order to generate one or more output values; during the forward propagation of the one or more first training data points, calculating, for at least a first of the synapses, a first co-activation value; generating, for at least the first synapse, a first weight gradient by backpropagating through the network derivatives of an error value generated based on the one or more output values; and selectively updating a synaptic weight of at least the first synapse based on the first co-activation value.