Spiking Neural Network Saturation Handling via Intermediate Neuron Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Spiking neural networks face challenges in accurately reflecting the dot product of input and fan-in weights due to saturation issues, where neurons become overwhelmed and unable to process additional input spikes effectively, leading to information loss.
Innovation Solution
A network restructuring mechanism is introduced, which adds intermediate layers of neurons and synapses, reducing the fan-in connections to output neurons below saturation levels, allowing for efficient signal transmission and maintaining accuracy through the use of a guiding equation to determine the necessary network parameters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If feed forward inhibition network is used to deal with saturation, then saturation of spiking neurons can be managed, but the number of synapses and neurons increases to O(n2) and 2n respectively
Solution Approach 1:
The patent segments the direct fan-in connections to output neurons by introducing intermediate neurons that divide the input processing into multiple stages. This segmentation reduces the fan-in of individual neurons from O(n) to O(sqrt(n)), thereby managing saturation while reducing the total number of synapses from O(n2) to O(n*sqrt(n))
Solution Approach 2:
The patent introduces an intermediate layer dimension between input and output neurons, transforming the direct connection architecture into a multi-layer structure. This dimensional addition allows signal distribution across multiple processing stages, reducing the computational burden on individual neurons and their synapses
2Reliability
If increase the number of time steps and use larger membrane time constants, then saturation can be avoided, but precision allowed by hardware is compromised
Solution Approach 1:
The patent applies different structural configurations to different parts of the network - intermediate neurons have specific fan-in limits while output neurons have reduced fan-in connections. This local structural optimization allows the network to handle saturation locally without requiring global changes that would compromise hardware precision
3Productivity
If original network structure is maintained, then hardware efficiency is preserved, but signal saturation occurs leading to information loss
Solution Approach 1:
The patent introduces intermediate neurons as mediators between input and output neurons. These intermediary elements process and redistribute input signals, preventing direct saturation of output neurons while maintaining efficient signal transmission. The intermediate layer acts as a buffer that preserves information without requiring excessive hardware resources
Data Source
AI summary
The present disclosure provides for generating a spiking neural network. Generating a spiking neural network can include determining that a first input fan-in from a plurality of input neurons to each of a plurality of output neurons is greater than a threshold, generating a plurality of intermediate neurons based on a determination that the first input fan-in is greater than the threshold, and coupling the plurality of intermediate neurons to the plurality of input neurons and the plurality of output neurons, wherein each of the plurality of intermediate neurons has a second input fan-in that is less than the first input fan-in and each of the plurality of output neurons has a third input fan-in that is less than the first input fan-in.