Self-adjusting Synaptic Threshold for Neural Network Activity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Artificial neural networks face challenges in achieving optimal neuronal activity and generalization due to fixed thresholds, requiring numerous parameters and additional processing time, especially in binary neural networks like Hierarchical Temporal Memory (HTM), where ensuring widespread neuronal activity is difficult without prior knowledge of input data statistics.
Innovation Solution
The introduction of a self-adjusting threshold mechanism in artificial neural networks, where an overlap value is determined for each neuron, and the activation threshold is dynamically adjusted to approach a predetermined neuronal activity target, ensuring a desired level of neuronal activity without requiring additional parameters or memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a fixed activation threshold is used in the neural network, then the network structure remains simple, but the neuronal activity distribution becomes suboptimal and requires numerous parameters for tuning
Solution Approach 1:
The patent implements a dynamic activation threshold that automatically adjusts based on the overlap between active inputs and synapses. Instead of using a fixed threshold, the system computes an overlap value for each neuron and uses this to dynamically determine activation, allowing the threshold to adapt to different input patterns and maintain optimal neuronal activity distribution without requiring manual parameter tuning
2Reliability
If the activation threshold is lowered to increase neuronal activity, then more neurons become active, but the network requires additional parameters and memory to manage the increased complexity
Solution Approach 1:
The system performs self-adjustment by computing overlap values for each neuron based on active inputs and synaptic weights, then using these overlap values to dynamically set activation thresholds. This self-service mechanism allows the network to automatically maintain appropriate neuronal activity levels without requiring external parameter tuning or additional memory structures to track activity statistics
3Manufacturing precision
If prior knowledge of input data statistics is used to set the activation threshold, then optimal neuronal activity can be achieved, but the system loses adaptability to different input distributions
Solution Approach 1:
The activation threshold is made dynamic and adaptive by computing it based on the actual overlap between currently active inputs and synaptic weights. This allows the threshold to automatically adapt to different input distributions and patterns without requiring prior knowledge or manual tuning, maintaining optimal neuronal activity across diverse input scenarios
Solution Approach 2:
The system changes the activation threshold parameter dynamically based on the computed overlap value. Instead of using a static parameter, the threshold is adjusted according to the specific input pattern being processed, allowing the network to adapt to different input statistics while maintaining precise control over neuronal activity
Data Source
AI summary
Self-adjusting thresholds for synaptic activity in neural networks are provided. In various embodiments, for each of a plurality of neurons within an artificial neural network, an overlap value is determined corresponding to active inputs connected to the neuron via synapses having non-zero synaptic weights. A count of those of the plurality of neurons whose overlap exceeds an activation threshold of the neural network is determined. The count is compared to a predetermined neuronal activity target. The activation threshold of the neural network is adjusted to approach the predetermined neuronal activity target.