Differentiable Spiking Neuron Model for SNN Training Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Spiking neural networks face inefficiencies in training due to their discontinuous and nonlinear mechanisms, particularly when dealing with high time-resolution inputs, leading to a large number of simulation time steps and inefficient training for complex tasks.
Innovation Solution
An automatic differentiable spiking neuron model and training method that allows multiple spikes in a single simulation time step, using weighted summation and threshold-based spike determination, along with a total loss function combining classification and activity losses for efficient training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a single-spike mechanism is used at each time step, then the spiking neural network can maintain its discontinuous and nonlinear mechanism, but the number of simulation time steps becomes extremely large and unacceptable for high time-resolution inputs
Solution Approach 1:
The patent segments the spike generation process by allowing multiple spikes to be generated within a single simulation time step. Instead of limiting the network to one spike per time step, the invention divides the time step into multiple opportunities for spike generation, effectively segmenting the temporal resolution without increasing the number of simulation steps required.
Solution Approach 2:
The patent introduces a new dimension to the spiking mechanism by allowing multiple spikes per time step. This transforms the traditional single-spike constraint into a multi-spike capability, adding a dimensional aspect to spike generation that enables handling high time-resolution inputs more efficiently without proportionally increasing simulation time steps.
2Ease of manufacture
If the conventional standard error backpropagation through time is used, then the training algorithm can be simple, but it is not directly compatible with the non-differentiable spiking generation function
Solution Approach 1:
The patent introduces an intermediary mechanism that bridges the gap between the non-differentiable spiking function and the gradient-based backpropagation algorithm. By using a surrogate gradient approach, the invention creates a differentiable approximation that allows standard backpropagation to work with spiking neural networks, maintaining algorithmic simplicity while achieving compatibility.
Solution Approach 2:
The patent modifies the training approach by changing the parameter representation and gradient computation method. Instead of directly differentiating the non-differentiable spiking function, the invention transforms the problem into one where gradients can be computed through parameter changes in the surrogate gradient, enabling compatibility with standard backpropagation.
3Productivity
If multiple spikes are allowed in one simulation time step, then the training efficiency is greatly improved, but the network complexity increases
Solution Approach 1:
The patent introduces dynamic spike generation where the number of spikes per time step is not fixed but determined dynamically based on the membrane potential and threshold conditions. This dynamic approach allows the network to adapt the complexity of spike generation to the input requirements, improving training efficiency without permanently increasing structural complexity.
Solution Approach 2:
The patent implements periodic spike generation within simulation time steps based on threshold crossing events. Instead of continuously generating spikes or using a fixed number, the system periodically evaluates the membrane potential against the threshold, generating spikes only when conditions are met. This periodic action improves efficiency by avoiding unnecessary computations while maintaining the capability for multiple spikes when needed.
Data Source
AI summary
A signal processing method for neurons in a spiking neural network is disclosed. The spiking neural network includes a plurality of layers, each of the layers includes a plurality of neurons, and the signal processing method includes following steps: a receiving step: at least one neuron configured to receive at least one path of input spike train, an accumulation step: performing weighted summation based on the at least one path of input spike train to obtain a membrane potential, and an activation step: when the membrane potential exceeds a threshold value, determining an amplitude of a spike fired by the at least one neuron based on a ratio of the membrane potential to the threshold value.