Neural Network Weight Update via Local Activity Differences
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current training methods for deep neural networks, such as back-propagation, are energy inefficient and biologically implausible, necessitating the development of more efficient and biologically plausible training techniques.
Innovation Solution
The proposed method, activity-difference training, uses locally available information within the learning network to identify weight changes without explicitly calculating errors, leveraging a free inference phase and a biased inference phase to update weights based on the difference between the two phases.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If back-propagation is used to train deep neural networks, then training can be performed with explicit error calculation and gradient propagation, but energy consumption increases significantly
Solution Approach 1:
The patent segments the training process into local updates at each neuron based on local activity differences, rather than performing global back-propagation through the entire network. Each neuron independently updates its weights using only local information from its inputs and outputs, eliminating the need for extensive gradient propagation and significantly reducing energy consumption while maintaining training effectiveness
Solution Approach 2:
The training method enables each neuron to perform self-learning by autonomously adjusting its own weights based on local activity differences between pre- and post-bias states. This self-service approach eliminates the need for centralized control and explicit error calculation across the entire network, reducing overall energy consumption while achieving convergence
2Productivity
If back-propagation is used to train deep neural networks, then weights can be updated using explicit gradient information, but the method is biologically implausible
Solution Approach 1:
Instead of propagating error gradients backward through the network as in back-propagation, the patent inverts the approach by using only forward-pass activity differences. Each neuron compares its pre-bias and post-bias output states locally, and weight updates are driven by these local activity differences rather than by back-propagated gradients, making the process biologically plausible while maintaining training productivity
Solution Approach 2:
The patent applies local quality by restricting weight updates to each individual neuron based solely on that neuron's local activity differences. No global error information is propagated; each neuron learns independently using only its own input-output activity changes, which aligns with biological learning mechanisms while preserving efficient training
3Reliability
If traditional training methods are used, then comprehensive error correction can be achieved, but computational complexity and time consumption increase
Solution Approach 1:
The training process is segmented into independent local updates at each neuron rather than a coordinated global optimization process. Each neuron performs its own weight adjustment based on local activity differences, which can proceed in parallel without sequential dependency, significantly reducing training time while maintaining convergence through iterative local optimizations
Solution Approach 2:
The patent applies partial action by performing weight updates based only on local activity differences rather than comprehensive global error analysis. This partial approach focuses computational resources on locally relevant information, reducing the computational burden and training time while still achieving sufficient convergence for practical applications
Data Source
AI summary
A method for performing learning is described. A free inference is performed on a learning network for input signals. The input signals correspond to target output signals. The learning network includes inputs that receive the input signals, neurons, weights interconnecting the neurons, and outputs. The learning network is described by an energy for the free inference. The energy includes an interaction term corresponding to interactions consisting of neuron pair interactions. The free inference results in output signals. A first portion of the plurality of weights corresponding to data flow for the free inference. A biased inference is performed on the learning network by providing the input signals to the inputs and bias signals to the outputs. The bias signals are based on the target output signals and the output signals. The bias signals are fed back to the learning network through a second portion of the weights corresponding to a transpose of the first portion of the weights. At locations in the learning network, learning network equilibrium states are determined for the biased inference. The weights are updated based on the learning network equilibrium states.