Static Training for Spiking Neural Networks

Resolve Bottlenecks,
Find Innovative Solutions
Generate Solutions

Solution Overview

Problem

Existing deep artificial neural networks (ANNs) struggle to implement biologically plausible spiking neurons efficiently, as previous approaches rely on non-biologically plausible models and assume multiple spiking neurons per node, limiting their flexibility and hardware implementation options.

Innovation Solution

A method is developed to design single or multi-layer spiking neural networks using statically trained networks with dynamic node response functions that exhibit brief temporal nonlinearities, allowing for the conversion of static networks to dynamically implemented spiking networks with minimal performance error, employing techniques like leaky integrate-and-fire neurons and noise training.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If multiple spiking neurons per node are used in deep networks, then biological plausibility is improved, but device complexity and efficiency worsen

Engineering Contradiction:
Improvebiological plausibilityVSAvoidnumber of spiking neurons per node
Core Design Contradiction:
ReliabilityVSDevice complexity

Solution Approach 1:

Instead of using multiple spiking neurons per node to achieve biological plausibility, the patent inverts the approach by using a single spiking neuron per node. The biological plausibility is maintained through the use of realistic neuron models (LIF, I&F, LIF with adaptation) and synaptic dynamics (STDP, rate-based learning) rather than through neuronal multiplication. This single-neuron-per-node design reduces device complexity while preserving biological realism through accurate modeling of neuronal and synaptic behavior.

Inventive Principle:
Principle #13The other way round (Inversion)

2Ease of manufacture

If integrate-and-fire neuron models are used, then implementation simplicity is improved, but biological plausibility worsens

Engineering Contradiction:
Improveimplementation simplicityVSAvoidbiological plausibility
Core Design Contradiction:
Ease of manufactureVSReliability

Solution Approach 1:

The patent employs multiple spiking neuron models with varying degrees of biological realism to balance implementation simplicity and biological plausibility. The Leaky Integrate-and-Fire (LIF) model provides a simplified implementation while the Adaptive LIF model with adaptation currents and the I&F model with detailed synaptic dynamics provide enhanced biological plausibility. This parameter-based approach allows selection of appropriate model complexity for different applications, maintaining ease of implementation while improving biological realism when needed.

Inventive Principle:
Principle #35Parameter changes

3Productivity

If static response functions are used for training, then training efficiency is improved, but dynamic implementation accuracy worsens

Engineering Contradiction:
Improvetraining efficiencyVSAvoiddynamic implementation accuracy
Core Design Contradiction:
ProductivityVSMeasurement precision

Solution Approach 1:

The patent uses static response functions during the training phase to achieve training efficiency, then applies these trained weights to dynamic spiking neuron implementations. The static training phase allows efficient computation using standard backpropagation, while the dynamic implementation phase captures temporal dynamics. This preliminary training approach maintains training efficiency while enabling accurate dynamic implementation through proper weight transfer and subsequent fine-tuning if needed.

Inventive Principle:
Principle #10Preliminary action

Data Source

PatentUS11126913B2Methods and systems for implementing deep spiking neural networks
Publication Date: 2021.09.21 APPL BRAIN RES INC
  • US11126913B2 patent drawing
  • US11126913B2 patent drawing
  • US11126913B2 patent drawing

AI summary

A method for implementing spiking neural network computations, the method including defining a dynamic node response function that exhibits spikes, where spikes are temporal nonlinearities for representing state over time; defining a static representation of said node response function; and using the static representation of the node response function to train a neural network. A system for implementing the method is also disclosed.