Spiking Neural Network Training Using Surrogate Gradients
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for training deep spiking neural networks (SNNs) face inefficiencies due to memory requirements increasing with time steps and deviating from biological online learning principles, leading to high memory and time complexity, especially when representing each neuron as a recurrent neural network.
Innovation Solution
The proposed solution involves a computing device with a spiking neural network (SNN) that includes a pulse generator, eligible potential generation circuits, and a loss circuit module, which calculates and backpropagates loss values efficiently using surrogate online learning at once (SOLO) algorithms, replacing activation functions with extended boxcar functions and using eligible potential generation circuits to propagate gradients spatially, thereby reducing memory and time complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If each neuron is represented as a recurrent neural network to capture temporal dynamics, then the modeling accuracy is improved, but the memory requirements and time complexity increase significantly
Solution Approach 1:
The patent extracts only the essential temporal information needed for SNN training by using surrogate gradient methods that approximate temporal derivatives without requiring full RNN unrolling. This extracts the necessary temporal dynamics while removing the excessive memory burden of storing all intermediate states across time steps.
Solution Approach 2:
Instead of forward-propagating through time in RNNs and storing all intermediate states, the patent inverts the approach by using backpropagation through time with surrogate gradients that compute temporal derivatives in reverse. This allows capturing temporal dynamics without the forward-pass memory storage requirement.
2Reliability
If traditional backpropagation through time is used for training SNNs, then gradient computation is achieved, but the training time and computational complexity increase
Solution Approach 1:
The patent changes the parameter representation by using surrogate gradient functions that approximate the non-differentiable spike activation with differentiable surrogates. This parameter transformation enables gradient computation without requiring extensive time-consuming exact derivative calculations through complex temporal unrolling.
Solution Approach 2:
The patent uses inexpensive surrogate gradient approximations instead of expensive exact gradient computations. These surrogate gradients are computationally cheap to calculate and provide sufficient training signal, sacrificing some precision for significant speed gains in the training process.
3Adaptability or versatility
If discrete spike signals are used for communication between neurons, then biological fidelity is improved, but the compatibility with standard deep learning frameworks decreases
Solution Approach 1:
The patent introduces surrogate gradient functions as intermediary elements that bridge discrete spike signals and continuous gradient computation. These surrogates act as mediators that translate the discrete biological-like spike events into continuous differentiable signals that can flow through standard deep learning frameworks, enabling both biological fidelity and framework compatibility.
Data Source
AI summary
A computing device and an operating method of the computing device based on a spiking neural network (SNN) are disclosed. The computing device includes a pulse generator configured to generate a pulse corresponding to an input spike signal; an SNN including layers of spiking neurons each generating an output spike signal by applying the pulse to a neuron model; and a loss circuit module configured to calculate a loss value based on a potential value accumulated by the output spike signal generated for each of the layers and backpropagate the loss value to the SNN.