Sleep-Inspired ANN to SNN Conversion for Memory Consolidation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Artificial neural networks (ANNs) suffer from catastrophic forgetting, limited generalization, and poor transfer learning, which hinders their ability to perform well on new tasks without forgetting older tasks.
Innovation Solution
A sleep-inspired algorithm that converts an ANN into a spiking neural network (SNN), applies an unsupervised spike-timing-dependent plasticity (STDP) phase with noisy input to simulate deep sleep dynamics, and then converts the SNN back to an ANN to enhance memory consolidation and generalization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If an ANN is trained with standard deep learning methods, then it achieves good performance on training tasks, but it suffers from catastrophic forgetting and poor generalization to new tasks
Solution Approach 1:
The patent applies spike-timing-dependent plasticity (STDP) learning rules during a sleep simulation phase before the ANN encounters new tasks. This preliminary action of consolidating memory representations through biologically-inspired learning mechanisms prepares the network to retain old task knowledge while adapting to new tasks, thereby preventing catastrophic forgetting and improving generalization capability
Solution Approach 2:
The patent transforms the ANN into a spiking neural network (SNN) by changing the activation and learning parameters to mimic biological neuron behavior. This parameter transformation enables the network to perform unsupervised STDP learning during sleep phases, which fundamentally alters how weights are updated and consolidates memory representations, leading to improved reliability and adaptability
2Adaptability or versatility
If an ANN is converted to a spiking neural network and applies unsupervised STDP phase with noisy input, then it improves memory consolidation and generalization, but it increases computational complexity and training time
Solution Approach 1:
The patent implements periodic sleep-wake cycles where the SNN alternates between unsupervised STDP learning phases (sleep) and supervised training phases (wake). This periodic structure allows the network to consolidate memory during sleep phases without continuously increasing computational complexity, as the STDP phase is applied in discrete intervals rather than continuously
Solution Approach 2:
The patent converts the ANN to an SNN, applies the sleep simulation with STDP learning, then converts the SNN back to an ANN format. This copying approach allows the beneficial memory consolidation effects to be transferred to the standard ANN architecture, enabling the complex SNN processing to be performed only temporarily during the sleep phase while maintaining compatibility with standard deep learning workflows
Data Source
AI summary
An example method of the presently disclosed technology may include: (1) transforming a neural network from an artificial neural network (ANN) to a spiking neural network (SNN); (2) when the neural network is transformed into the SNN, modifying synaptic weights of the neural network by applying a simulated memory replay process to the neural network; and (3) after applying the simulated memory replay process to the neural network, transforming the neural network from the synaptic weight-modified SNN to a synaptic weight-modified ANN.