Differentiable Temporal Point Process for Spiking Neural Network Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods face difficulties in performing gradient descent of the Evidence Lower BOund (ELBO) for temporal and partially-observable multivariate point processes, particularly in spiking neural networks, due to the lack of differentiability in existing approaches.
Innovation Solution
A differentiable temporal point process is developed, allowing for the generation of samples using a Poisson process and a concrete distribution, enabling the output to be differentiable with respect to model parameters, which facilitates gradient descent and training of probabilistic spiking neural networks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If current methods are used for temporal point processes, then the model can be implemented, but gradient descent of ELBO cannot be performed due to lack of differentiability
Solution Approach 1:
The patent introduces an intermediary sampling process that bridges the non-differentiable Poisson process and the differentiable concrete distribution. The concrete distribution acts as a mediator that approximates the discrete spike events while maintaining differentiability with respect to model parameters, enabling gradient-based optimization.
Solution Approach 2:
The patent changes the parameterization approach by introducing a temperature parameter τ that controls the softness of the concrete distribution. By varying this parameter, the model transitions between differentiable soft assignments and discrete hard assignments, enabling gradient flow during training while maintaining compatibility with spiking neural network semantics.
2Measurement precision
If discrete sampling is used, then the point process is accurate, but variance in gradient estimates increases
Solution Approach 1:
The patent applies partial action by using soft assignments instead of complete discrete assignments. The concrete distribution provides partial assignments to multiple potential spike events, which reduces the variance in gradient estimates while still capturing the essential discrete nature of spiking events through the temperature parameter.
3Adaptability or versatility
If conventional neural networks are converted to SNNs, then compatibility with neuromorphic chips is achieved, but information loss occurs during conversion
Solution Approach 1:
The patent performs preliminary action by designing the spiking neural network model from the ground up to be differentiable, rather than converting conventional networks afterward. The differentiable point process is integrated into the network architecture before deployment, preserving information while ensuring compatibility with neuromorphic hardware through the spiking event formulation.
Data Source
AI summary
A method performs a Differentiable Point Process (DPP). Generate a first sample sk by sampling from a Poisson process with reference to an upper bound λ of a conditional intensity function representing the DPP given a first set of samples S. Determine whether sk>T, output a second set of samples and train a probabilistic model using when sk>T, and perform the next four steps (generate a second sample, add, add, update) and return to the first step (generate a first sample) when sk≤T, where T denotes an observation length. Generate a second sample[pkrk]by sampling from a concrete distribution with reference to a parameter of the distribution defined by the conditional intensity function and a temperature τ, given a second set of samples . Add a pair of sk and pk to and discard rk. Add sk to S. Update k to k+1.