Neural Network Processor Hysteresis Spiking With Minimal Neuron Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Direct implementation of artificial neural networks without considering data sparsity leads to high processing power consumption and scalability issues, and existing solutions that mitigate these issues require significant storage space for each neuron.
Innovation Solution
A neural network processor with state registers for each element that transmits differential event messages based on state changes exceeding a threshold, using a binary indicator to reduce storage requirements and computation load by selectively generating output messages.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Object-generated harmful factors
If a hysteresis quantizer is used to mitigate excessive spiking, then spiking activity is reduced, but storage space per neuron increases significantly
Solution Approach 1:
The patent extracts only the essential information needed for hysteresis decision-making (the quantized output of the previous time step) and separates it from the full state storage requirement. By storing only the previous quantized output value rather than the complete previous state, the system achieves hysteresis functionality with minimal storage overhead, resolving the contradiction between reducing excessive spiking and maintaining storage efficiency.
Solution Approach 2:
The patent applies local quality by making the storage requirement local and selective rather than global and comprehensive. Instead of storing all state information for each neuron, only the specific previous quantized output value needed for hysteresis comparison is stored. This localized storage approach enables hysteresis-based spiking reduction while keeping storage requirements per neuron minimal.
2Speed
If direct quantization is applied to neural network outputs, then processing speed is improved, but excessive firing activity occurs
Solution Approach 1:
The patent implements feedback by comparing the current quantized output with the previous quantized output stored in memory. This feedback mechanism enables hysteresis detection, where the system remembers the previous state and only triggers a spike when the quantized output changes in a sustained manner rather than due to small oscillations. This feedback-based approach maintains processing speed while eliminating excessive firing activity.
Solution Approach 2:
The patent applies preliminary action by pre-storing the quantized output value from the previous time step before the current quantization decision is made. This preliminary storage of previous state information enables the hysteresis comparison to be performed efficiently during the current processing cycle, maintaining high processing speed while preventing excessive spiking through proactive state memory.
3Measurement precision
If high processing power is used for direct neural network implementation, then accuracy is maintained, but energy consumption increases vastly
Solution Approach 1:
The patent changes the parameter of quantization from direct full-precision quantization to hysteresis-based quantization that incorporates temporal information. By modifying the quantization parameter to include memory of previous states, the system achieves the same or better effective precision while dramatically reducing computational requirements and energy consumption through the selective triggering of spike events based on sustained changes rather than instantaneous variations.
Data Source
AI summary
A neural network processor is provided comprising a plurality of mutually succeeding neural network processor layers is provided. A neural network processor layer therein comprising a plurality of neural network processor elements (1) having a respective state register (2) for storing a state value (X) indicative for their state, as well as an additional state register (4) for storing a value (Q) of a state value change indicator that is indicative for a direction of a previous state change exceeding a threshold value. Neural network processor elements in a neural network processor layer are configured to selectively transmit differential event messages indicative for a change of their state, dependent both on the change of their state value and on the value of their state value change indicator.