Time-weighted synapse-based digital neuromorphic co-processor

Abstract

The present application describes a neuromorphic co-processor suitable for deployment onto FPGA, ASIC and similar technologies. The Neuromorphic co-processor is characterized by comprising a range of sequentially chained together neurons in a spiking neural network like arrangement, each of said neurons comprising at least an input and an output, the input being configured to acquire a range of time-weighted signals, and the output being configured to provide an output spike signal, wherein the output spike signal is determined by a threshold module which is connected to a threshold membrane fed by the input.

Description

Time-Weighted Synapse-Based Digital Neuromorphic Co-Processor

[0001] The present application describes a neuromorphic co-processor suitable for deployment onto FPGA, ASIC and similar technologies.

[0002] Spiking Neural Networks (SNNs) are artificial neural network models that closely mimic biological neuron structures, when compared to conventional Artificial Neural Networks (ANNs) – such as Deep Neural Networks (DNNs), Conventional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) - which have been widely adopted in academic research and industrial applications.

[0003] When discussing SNNs, it is useful to discuss how they differ from other more conventional ANNs to better understand the advantages and disadvantages of this type of algorithm, as both are used to handle similar tasks in different ways.

[0004] Both conventional ANNs and SNNs require an initial training stage, where a dataset representative of the desired application must be provided. During training, internal model weights and parameters are adjusted until a desired level of accuracy is attained, in an iterative approach. Due to their increased temporal complexity, SNN training is more computationally intensive and time consuming. Like their biological counterparts, SNNs and conventional ANNs must grow in size and complexity to solve more intricate problems – this has led to an increase in training and inference costs, and motivated researchers to find solutions to alleviate both tasks.

[0005] The differences between conventional ANNs and SNNs are starker during the inference or deployment stage. In this phase, the algorithm is fed new input data, with the hope of retrieving accurate output data once computation is finished.

[0006] Firstly, conventional ANNs are used to process multidimensional input arrays, known informally as tensors – these could be images, text characters, sound, sensor data, among others. SNNs may also process this type of data, but they require that it be converted to a spike train. This encoding process can take on many forms based on the application at hand, where both amplitude and frequency domain encoding methods are possible.

[0007] The same holds true for the output data produced by these algorithms. Conventional ANNs generate output tensors, while SNNs generate one or more output spike trains. Therefore, a decoding stage may also be required to interpret and expose data externally in a more conventional format.

[0008] Finally, the internal arrangement of conventional ANNs and SNN also differ substantially. Conventional ANNs are composed internally by one or more layers, with one or more of these layers performing the convolution operation. Data flows in a single direction during inference, from the first layers to the last, generating a uniform computational workload regardless of the input data provided. While conventional ANNs are inspired by biological processes, particularly the visual cortex, low-level components such as neurons and synapses are not replicated in its internal architecture. For SNNs, the model used to represent neuron and synapse behavior can vary depending on the biological accuracy required, and the available computational resources. In practice, the more basic neuron models are favored over the more biologically accurate and complex ones, as they enable sufficient accuracy with less computational effort. The layer structure and forward-propagation of data are common characteristics of both architectures. However, the computation associated with any given neuron is only carried out if it has received a stimulus (input spike) from preceding layers. For this reason, SNN inference is inherently event-based and sparser when compared to conventional ANNs inference.

[0009] Compared to their more mainstream counterpart, SNNs have highly desirable characteristics for deployment onto programmable or custom hardware devices. However, they are not well suited for general purpose computing devices, as these are not capable of leveraging the unique advantages of SNNs – in fact, using conventional ANNs may indeed offer better performance using conventional hardware. For this reason, SNNs can be deployed onto specialized neuromorphic co-processor devices, accelerating a demanding SNN inference task, relieving any processing unit to carry out other tasks in the meantime. The use of domain-specific processors to accelerate the training task is done for both conventional ANNs and SNNs but is out of scope of this document.

[0010] Once again, it is useful to examine conventional ANNs and their computational requirements as a preamble to a discussion of SNN hardware accelerator architectures. The most computationally intensive task of conventional ANNs inference is the convolution operation, which may be accelerated by architectures with numerous multipliers operating in parallel. Other layers, notably the fully connected layer, are often constrained by memory bandwidth. With the recent growth in conventional ANNs complexity, hardware accelerators for these models must also provide larger embedded memories to store weights and intermediate results.

[0011] Their event-driven nature makes them well suited for consuming data generated in an event-driver manner, such as event cameras. Their sparse computation characteristics may also lead to reduced power consumption and lower latency. Finally, several SNN implementations have been proposed that preclude the use of multipliers entirely, instead relying on adders and carry chains to compute neural activity – this is advantageous as adders require much less chip area and power compared to multipliers. However, this perceived advantage is moot when deploying on GPUs, as the large multiplier structures go unused. In summary, SNN possess a set of desirable characteristics that are in fact very difficult to leverage with traditional CPU and GPU devices – instead, configurable HW devices such as FPGAs and ASICs are much better suited, and the focus of this application.

[0012] The present invention describes a Neuromorphic co-processor characterized by comprising a range of sequentially chained together neurons in a spiking neural network like arrangement, each of said neurons comprising at least an input and an output, the input being configured to acquire a range of time-weighted signals, and the output being configured to provide an output spike signal, wherein the output spike signal is determined by a threshold module which is connected to a threshold membrane fed by the input.

[0013] In a proposed embodiment of present invention, the threshold membrane acquires the input with a range of time-weighted signals and actuate the threshold module once the threshold membrane reaches a pre-defined threshold value defined in a parameter input of the threshold module, said threshold module generating an output spike signal equal to a timestep of the range of time-weighted signals and also a trigger to a refractory time module to enable the reset of the neuron.

[0014] Yet in another proposed embodiment of present invention, the refractory time module is connected within the output of the threshold module and an entry of an enable module which provides an input data to control the threshold membrane.

[0015] Yet in another proposed embodiment of present invention, the threshold membrane further comprises an input data from a reset state module.

[0016] Yet in another proposed embodiment of present invention, the time-weighted signals are provided by a synapse module external to the neuron and / or other neurons in preceding layers in a feed-forwards network, the time-weighted signal being a set neuron processes which comprise a range of input spikes which will processed by the Synapse module which will convert it into the time-weighted signal based on a weight parameter defined in the Synapse module.

[0017] Yet in another proposed embodiment of present invention, the Neuromorphic co-processor is characterized by being deployed in a FPGA, ASIC or a similar hardware device.

[0018] Yet in another proposed embodiment of present invention, the neuron is suitable to perform like an accumulator, incrementing for every time-weighted signal input received, starting from a reset state, and once the predefined threshold value is reached, the neuron generates the output spike with a duration equal to one timestep, and simultaneously returning to a reset state maintained for a set amount of execution timesteps defined by the refractory time.

[0019] Yet in another proposed embodiment of present invention, the input spikes comprise a range of data input types like text, images, audio, video or other digital media.

[0020] Yet in another proposed embodiment of present invention, the synapse module comprises N binary signal inputs for collecting one or more input spikes, one for each neuron of the preceding neuron layer when using fully connected layers, wherein these binary signal inputs are active-high signals, being HIGH for a clock cycle at the time; and by comprising weight parameter which sets a configuration value which defines the number of execution timesteps of the time-weighted signal.

[0021] The present invention further discloses a computer program, characterized by comprising programming code or instructions suitable for enabling the operating functions of the Neuromorphic co-processor described in any of the preceding claims, in which said computer program is stored, and is executed in a data processing system, remote or in-site, for example a server, performing the respective steps described in the claims.

[0022] The invention still discloses a computer readable physical data storage device, in which the programming code or instructions of the computer program are stored.General Description

[0023] The current application discloses a novel implementation of a neuromorphic co-processor suitable for deployment onto FPGA, ASIC and similar technologies. A specific adaptation allows for simplification of hardware realizations of the architecture, leading to reduced hardware resource usage, lower cost, lower power consumption and greater configurability.

[0024] Generically speaking, a herein proposed neuromorphic co-processor is a specialized processing unit designed to mimic the structure and function of the human brain’s neural networks. The neuromorphic co-processor is to be implemented built using neuromorphic computing principles, but the architecture is based on how biological neurons and synapses operate, aiming to provide efficient computation for tasks like pattern recognition, real-time learning, and sensory data processing, typically with much lower power consumption compared to traditional CPUs or GPUs.

[0025] The core innovation lies in the time modulation of synaptic connections between neurons (sometimes informally referred to as ‘weights’). This adaptation lends itself well to fully or partially asynchronous implementations. More crucially, it precludes the use of large multi-operand adder structures which require a lot of chip area and power to implement – an issue exacerbated in fully connected neuron layers seen in conventional ANNs.

[0026] Implementations using amplitude modulation of synaptic connections map poorly onto FPGA and ASIC architectures, particularly when considering asynchronous or partially asynchronous designs. More specifically, representing synaptic connections (weights) as a multiple bit numeric value requires the use multi-operand adder structures to compute membrane potential at each execution step. These require a lot of chip area and power to implement. It is possible to time-multiplex a smaller adder structure with fewer operands, paired with an accumulator, which requires more control logic and more execution cycles. More critically, it is not feasible to implement a fully asynchronous neuromorphic co-processor of SNNs under these constraints.

[0027] Implement synaptic connections (weights) between neurons using time modulation. For instance, represent stronger connections using longer duration pulses, and weaker connections with shorter pulses. The proposed arrangement maintains the core structure of the SNN, assuming that the integrate-and-fire neuron model or similar is being used. This adaptation allows for mathematically equivalent implementations of a base SNN structure; thus, it does not require re-training the network.

[0028] The advantages materialize when considering the new range of options available for hardware implementations in suitable technologies, more specifically FPGAs and ASICs. Typically, weights are represented as signed or unsigned integer, fixed-point or floating-point value. For a synchronous implementation, the computation of membrane potential at each execution step requires the use of an adder structure. Given the possibility of several incoming spikes from different neurons arriving at the same time, it is necessary to either: a) implement a large, multiple-operand adder which is expensive in both chip area, resources and power, or b) multiplex a reduced operand adder over several execution cycles, increasing latency. The proposed novelty allows for binary spike trains with time modulation of weights, meaning that membrane potential calculation (i.e., the N-operand adder) can be done with combinational logic in a single timestep (for instance, by using a Look-up Table). The advantages are even more substantial in a fully asynchronous design, as combinational logic is much more straightforward to implement and validate when compared to an asynchronous adder, let alone one with numerous operands. The proposed novelty leads to implementations that more closely resemble their biological counterparts, particularly in the ASIC case, which could be advantageous for those interested in replicating real organisms as close as possible.

[0029] For better understanding of the present application, figures representing preferred embodiments are herein attached which, however, are not intended to limit the technique disclosed herein.

[0030] Illustration of the integrate and fire model adopted for the neuron (203) behavior in this embodiment, using binary spike trains and time-weighted synaptic connections. The neuron receives several input spikes (80), coming from other neurons (203) in preceding layers in a feed-forwards network. Each incoming spike (80) is processed by a Synapse (time weighting) module (201), which converts it into a time-weighted signal (91) based on the weight parameter (202). This time-weighted signal (91) is then set as an input of the neuron process. The neuron module (203) acts as an accumulator, incrementing for every time-weighted signal (91) input received, starting from a reset state (209) which is assumed to be zero in this example. Once a certain predefined threshold value (204), greater than zero is reached, the neuron (203) generates an output spike (205) whose duration equals one timestep. As this occurs, the neuron (203) returns to its reset state (209), which is maintained for a set amount of execution timesteps – defined as refractory time (208).

[0031] This Figure replicates the diagram of, with additional details provided on the inner components of the neuron module (203). In this image, the threshold membrane (206) acts as an event counter, incrementing for every execution timestep during which the time weighted signal input (91) is active. Once the threshold membrane (206) reaches a pre-defined threshold value (204), it actuates the threshold module (207). Once actuated, the threshold module (207) generates an output spike (205) and triggers the refractory time module (208) as previously described. The refractory time module (208) ensures the entire module (203) stays in its reset state for a specific amount of execution timesteps, during which no output spikes (205) are generated, and the value contained in the threshold membrane (206) is the one that corresponds to its reset state (209).

[0032] With reference to the figures, some embodiments are now described in more detail, which are however not intended to limit the scope of the present application.

[0033] Firstly, this embodiment considers a synchronous implementation in which the unit of time is the ‘execution timestep’. When working with SNNs, time is often discretized into equal, non-divisible units of time called timesteps.  A SNN may be implemented in many ways with different underlying hardware technologies,requiring more or less execution clock cycles to perform the associated computations – however, in the context of execution timesteps, all implementations should be equivalent.

[0034] As previously stated, the invention requires that input data is encoded in binary spike trains (80). For common digital data input types such as text, images, audio or others, a pre-processing mechanism will have to be used to obtain this feature. A common method to achieve this is through a Poisson distribution encoder, which will be configured to generate a pseudo-random or truly random sequence of interspaced binary spikes (80) based on the magnitude of the data input signal. This Poisson encoder may also be implemented in various forms depending on the available resources, with a particular desired level of randomness and characteristics of the digital input data. A common and simple method to generate random or pseudorandom numbers from common digital data input types required for a Poisson distribution encoder could leverage a Linear Feedback Shift Register – LSFR (110). Naturally, other means of describing the temporal distribution of spikes may be used for different embodiments of the proposed idea.

[0035] A more detailed description of this embodiment of the proposed arrangement follows, in which a Poisson distribution encoder is implemented. The suggested encoder based on a LSFR (100) consists of a series of D-type flip-flops connected in a linear sequence with feedback paths, which perform exclusive-or (XOR) operations on certain bits of the register to determine the output bit. To generate a random number within a wide range, a greater number of bits should be used. The N bit shift register is a sequence of D-type flip-flops connected in sequence – in the embodiment, each of these represent one bit of the final N-bit digital output word. Through a feedback circuitry, some XOR gates are used to shuffle the inputs of the N bit shift register in every clock cycle. The XOR gates take inputs from specific flip-flops outputs and combines them to form the feedback input. In the proposed embodiment, the LFSR will cycle through a sequence of pseudo random numbers which eventually repeat, starting from the first. The output of the random number generator, which is comprised by the N bit shift register and the XOR gates, is compared with an input value at every clock cycle – the hardware structure used in the present embodiment to achieve this is called the pixel-value comparator. The pixelvalue comparator performs a simple operation: If the input value is greater than the pseudo random value, it will generate an output signal ‘1’ logic, if not it will generate a ‘0’ logic. Therefore, it becomes evident that greater pixel values inputs are more likely to generate spikes on the output. The actual ‘spike generation’ task is performed by the spike generator module, but the decision to generate or not is performed by the pixel-value comparator. In this embodiment, all spikes are equal as they exit the spike generator.

[0036] The synapse module (201) allows the connection between neurons (203) and is illustrated in bothand 2. One or several random or pseudo-random spike trains (80) are the main inputs of the synapse module (201). The synapse module (201) modifies each incoming input spike, providing a temporal dimension to each one which is based on the desired “strength” of each synaptic connection – defined in the weight parameter (202). The synapse module (201) in the proposed embodiment has two digital inputs:

[0037] 1) One or more spike train inputs (80): the synapse module (201) contains N of these binary signal inputs, one for each neuron (203) of the preceding neuron layer when using fully connected layers. Considering that these are active-high signals, each only stays HIGH for a clock cycle at the time. The period between HIGH execution timesteps must exceed the ‘weight parameter’ value (202).

[0038] 2) weight parameter (202): a configuration value set at synthesis time (for an FPGA embodiment of the proposed arrangement) or during chip fabrication (for an ASIC-based embodiment). Alternatively, this value may indeed be modified during runtime when the neuromorphic co-processor is not under operation (i.e., during setup). This value defines the number of execution timesteps of the output pulse (91).

[0039] In each neuron (203), the neuron membrane (206) acts as a binary counter, which increments for every clock cycle in which the time-weighted input signal (91) is active. The threshold module (207) acts as a simple comparator, which activates then the value held by the threshold membrane (206) equals, or high-up, the threshold value (204). This threshold value should be set during FPGA synthesis or ASIC fabrication of the architecture, or even during a configuration stage at runtime – in any case, when synaptic / neuron activity is underway, it must remain constant in the proposed embodiment. If the described condition is met, the neuron (203) enters the “fully stimulated” state, and the threshold module (207) will generate an output pulse of 1 clock cycle. Simultaneously, the value contained in the threshold membrane (206) is reset, returning the entire neuron module (203) to its initial state. Furthermore, the refractory time (208) block ensures that the neuron (203) remains in this state for a set amount of execution timesteps, during which no output pulses are generated, and the value of the threshold membrane (206) remains unchanged.

[0040] The last layer of an artificial neural network is of particular significance, as it encodes the final classification of the input data once computing is concluded. The same hold true for the proposed embodiment. After a previously defined number of execution timesteps, the neuron (203) of the last layer that has achieved the ’fully stimulated’ state the most is considered the ‘winner’, i.e., the input data can be classified as the label associated with that neuron. Thus, the proposed arrangement is intended to perform a generic ‘data classification’ task. In this embodiment we consider fully connected network layers, i.e., all neurons of a given layer are connected to all neurons of the next layer, and to all neurons of the preceding layer.

[0041] The herein proposed method for time-based modulation of synaptic connections between neurons may be implemented using one or more processing units, one or more processing devices, any means for processing, such as a processor, a computer or a programmable hardware component being operable with accordingly adapted software. The key benefits of the proposed arrangement (described in prior sections), however, may be leveraged to a greater extent by architectures with a sufficient degree of hardware configurability – examples of these include FPGAs, ASICs and CPLDs.Examples

[0042] The herein disclosed neuromorphic co-processor arrangement can be used in areas like:

[0043] - Robotics: Real-time learning and adaptation for navigation and decision-making.

[0044] - AI at the edge: Processing sensory inputs (audio, video, touch) locally on low-power devices.

[0045] - Healthcare: Wearable devices for health monitoring and brain-machine interfaces.

[0046] - Autonomous systems: Efficient processing for self-driving cars and drones.

[0047] The proposed arrangement is an improvement upon existing neuromorphic co-processor architectures, providing advantages for HW-centric realizations of this type of algorithm. As such, it has wide-ranging industrial applicability, including but not limited to manufacturing, logistics, energy, and healthcare. It is particularly useful for emerging applications that rely on artificial intelligence with high computational and / or memory requirements. Due to its high similarity to biological neural networks, it is suitable for neurophysiological research.

[0048] 10 – Shift register N bits;

[0049] 20 – XOR gates;

[0050] 30 – Input;

[0051] 40 – feedback;

[0052] 45 - pseudoRandom output;

[0053] 50 – pixelvalue comparator;

[0054] 60 – Input Value;

[0055] 70 – Spikes generator;

[0056] 80 - spike output signal;

[0057] 91 – time-weighted signal;

[0058] 100 – Random Number Generator;

[0059] 110 – Linear Feedback Shift Register (LSFR);

[0060] 201 – Synapse (time weighting) module;

[0061] 202 – weight parameter;

[0062] 203 – Neuron;

[0063] 204 – threshold value;

[0064] 205 – one clock cycle;

[0065] 206 – threshold membrane;

[0066] 207 – threshold module;

[0067] 208 – Refractory time;

[0068] 209 – Reset state module;

[0069] 210 – Enable.

Claims

Neuromorphic co-processor characterized by comprising a range of sequentially chained together neurons (203) in a spiking neural network like arrangement, each of said neurons (203) comprising at least an input and an output, the input being configured to acquire a range of time-weighted signals (91), and the output being configured to provide an output spike signal (205), wherein the output spike signal (205) is determined by a threshold module (207) which is connected to a threshold membrane (206) fed by the input.Neuromorphic co-processor according to the previous claim, characterized by the threshold membrane (206) acquiring the input with a range of time-weighted signals (91) and actuating the threshold module (207) once the threshold membrane (206) reaches a pre-defined threshold value (204) defined in a parameter input of the threshold module (207), said threshold module (207) generating an output spike signal (205) equal to a timestep of the range of time-weighted signals (91) and also a trigger to a refractory time module (208) to enable the reset (209) of the neuron (203).Neuromorphic co-processor according to any of the previous claims, characterized by the refractory time module (208) being connected within the output of the threshold module (207) and an entry of an enable module (210) which provides an input data to control the threshold membrane (206).Neuromorphic co-processor according to any of the previous claims, characterized by threshold membrane (206) further comprising an input data from a reset state (209) module.Neuromorphic co-processor according to any of the previous claims, characterized by the time-weighted signals (91) being provided by a synapse module (201) external to the neuron (203) and / or other neurons (203) in preceding layers in a feed-forwards network, the time-weighted signal (91) being a set neuron processes which comprise a range of input spikes (80) which will processed by the Synapse module (201) which will convert it into the time-weighted signal (91) based on a weight parameter (202) defined in the Synapse module (201).Neuromorphic co-processor according to any of the previous claims, characterized by being deployed in a FPGA, ASIC or a similar hardware device.Neuromorphic co-processor according to any of the previous claims, characterized by the neuron (203) being suitable to perform like an accumulator, incrementing for every time-weighted signal (91) input received, starting from a reset state (209), and once the predefined threshold value (204) is reached, the neuron (203) generates the output spike (205) with a duration equal to one timestep, and simultaneously returning to a reset state (209) maintained for a set amount of execution timesteps defined by the refractory time (208).Neuromorphic co-processor according to any of the previous claims, characterized by input spikes (80) comprise a range of data input types like text, images, audio, video or other digital media.Neuromorphic co-processor according to any of the previous claims, characterized by the synapse module (201) comprising N binary signal inputs for collecting one or more input spikes (80), one for each neuron (203) of the preceding neuron layer when using fully connected layers, wherein these binary signal inputs are active-high signals, being HIGH for a clock cycle at the time; and by comprising weight parameter (202) which sets a configuration value which defines the number of execution timesteps of the time-weighted signal (91).Computer program, characterized by comprising programming code or instructions suitable for enabling the operating functions of the Neuromorphic co-processor described in any of the preceding claims, in which said computer program is stored, and is executed in a data processing system, remote or in-site, for example a server, performing the respective steps described in the claims.Computer readable physical data storage device, in which the programming code or instructions of the computer program are stored.

Images