Spiking Neural Network Back-Propagation Using Binary Error Encoding

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Spiking neural networks face challenges in implementing back-propagation algorithms due to hardware constraints, particularly requiring floating-point number multiplications, which is inefficient and not suitable for devices with limited resources, and existing solutions either require expensive implementations or do not account for asynchronous processing.

Innovation Solution

A new implementation of an error back-propagation algorithm using binary or ternary encoding to adapt to the constraints of spiking neural networks, allowing the same propagation infrastructure to be used for both data and error propagation, and incorporating modules for error calculation, integration, and synaptic weight updates without floating-point multiplications.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If conventional back-propagation algorithm is used, then training efficiency is improved, but hardware resource consumption increases due to floating-point multiplications

Engineering Contradiction:

Improvetraining efficiencyVSAvoidhardware resource consumption

Core Design Contradiction:

ProductivityVSUse of energy by moving object

Solution Approach 1:

The patent changes the parameter representation from floating-point numbers to binary or ternary encoded values. Errors are quantized and represented using discrete symbols (e.g., +1, 0, -1) instead of continuous floating-point numbers, enabling efficient computation with limited hardware resources while maintaining training effectiveness

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The patent replaces the mechanical multiplication operation with simpler computational elements. By substituting floating-point multiplication with binary/ternary arithmetic operations (AND, OR, NOT gates), the system achieves back-propagation without requiring complex floating-point multiplication hardware, thus reducing resource consumption

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Measurement precision

If floating-point number multiplications are used, then back-propagation calculation accuracy is improved, but device complexity increases

Engineering Contradiction:

Improvecalculation accuracyVSAvoiddevice complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent uses simple, inexpensive binary or ternary arithmetic operations instead of complex floating-point multiplication units. These discrete operations can be implemented with basic logic gates and accumulators, significantly simplifying the hardware architecture while providing sufficient calculation accuracy for neural network training

Inventive Principle:

Principle #27Cheap short-living objects (Disposable)

Solution Approach 2:

The patent segments the error representation into discrete quantized levels (binary or ternary values) rather than using continuous floating-point numbers. This segmentation allows the system to approximate error values using simple discrete arithmetic operations, reducing computational complexity while maintaining adequate precision for learning updates

Inventive Principle:

Principle #1Segmentation

3Speed

If asynchronous data processing is implemented, then processing speed is improved, but error propagation synchronization becomes difficult

Engineering Contradiction:

Improveprocessing speedVSAvoidsynchronization time

Core Design Contradiction:

SpeedVSLoss of time

Solution Approach 1:

The patent implements periodic error propagation cycles that synchronize with the asynchronous spike generation rhythm. Errors are propagated in discrete time steps corresponding to the spiking events, allowing the system to maintain synchronization without requiring continuous timing adjustments. The error back-propagation occurs periodically at the same rhythm as the forward propagation, ensuring temporal alignment

Inventive Principle:

Principle #19Periodic action

Solution Approach 2:

The patent prepares error signals in advance by quantizing and encoding them before propagation. By pre-processing and preparing error representations in the same asynchronous time steps as spike generation, the system eliminates synchronization delays. The error signals are ready for immediate propagation without requiring additional timing alignment operations

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS12147902B2Backpropagation of errors in pulsed form in a pulsed neural network

Publication Date: 2024.11.19 COMMISSARIAT A LENERGIE ATOMIQUE ET AUX ENERGIES ALTERNATIVES

AI summary

A new implementation is provided for an error back-propagation algorithm that is suited to the hardware constraints of a device implementing a spiking neural network. The invention notably uses binary or ternary encoding of the errors calculated in the back-propagation phase to adapt its implementation to the constraints of the network, and thus to avoid having to use floating-point number multiplication operators. More generally, the invention proposes a global adaptation of the back-propagation algorithm to the specific constraints of a spiking neural network. In particular, the invention makes it possible to use the same propagation infrastructure to propagate the data and to back-propagate the errors in the training phase. The invention proposes a generic implementation of a spiking neuron that is suitable for implementing any type of spiking neural network, in particular convolutional networks.