Selective Synaptic Plasticity Prevents Catastrophic Forgetting in Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Artificial neural networks suffer from catastrophic forgetting, rapidly forgetting previously learned tasks when presented with new training data due to uniform plasticity, which hinders their ability to learn new tasks without losing old knowledge.
Innovation Solution
Implementing a contrastive excitation backpropagation algorithm to identify important neurons and synapses for a task, and using Hebbian or Oja's learning rules to rigidify these connections, allowing them to maintain previously learned tasks while learning new ones through selective synaptic plasticity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If uniform plasticity is applied to all synapses in the artificial neural network, then the network can learn new tasks efficiently, but previously learned tasks are rapidly forgotten due to catastrophic forgetting
Solution Approach 1:
The patent applies local quality by differentiating synapses into important and unimportant categories based on their contribution to previously learned tasks. Important synapses are identified through a importance calculation mechanism and are protected from updating during new task learning, while unimportant synapses are allowed to update freely. This local differentiation resolves the contradiction by preserving critical knowledge in specific synapses while allowing adaptability in others.
Solution Approach 2:
The patent segments the synapse population into distinct groups (important and unimportant) based on their functional importance to previously learned tasks. This segmentation is achieved through calculating synapse importance metrics and applying selective update rules. By dividing the synapse set into protected and adaptable segments, the system maintains both stability of learned knowledge and plasticity for new learning.
2Productivity
If all synapse weights are updated during training on new tasks, then learning efficiency is improved, but catastrophic forgetting occurs
Solution Approach 1:
The patent implements local quality by applying different update rules to different synapses based on their importance. Important synapses (identified through importance calculation) maintain fixed weights to preserve reliability of previously learned tasks, while unimportant synapses undergo weight updates to improve learning efficiency for new tasks. This localized differential treatment resolves the contradiction between productivity and reliability.
Solution Approach 2:
The patent applies preliminary action by pre-identifying and protecting important synapses before the catastrophic forgetting occurs. The importance calculation is performed beforehand, and protected synapses are marked for exclusion from updates during new task training. This proactive protection mechanism prevents reliability degradation before it happens while allowing efficient learning in unprotected synapses.
3Adaptability or versatility
If the artificial neural network is uniformly plastic, then it can adapt to different tasks, but it loses previously acquired knowledge when presented with new training data
Solution Approach 1:
The patent resolves this contradiction by applying local quality through selective synapse protection. Each synapse is evaluated for its importance to previously learned tasks, and important synapses are granted stability (fixed weights) while unimportant synapses maintain plasticity (updateable weights). This creates a heterogeneous structure where stability and adaptability coexist in different parts of the network, preventing knowledge loss while preserving task adaptability.
Solution Approach 2:
The patent implements dynamics by making the plasticity property of synapses conditional rather than uniform. The importance metric dynamically identifies which synapses should be protected, and this protection status is applied dynamically during task transitions. This dynamic approach allows the network to adapt its stability-plasticity balance based on task requirements, maintaining both knowledge stability and task versatility.
Data Source
AI summary
An autonomous navigation system for a vehicle includes a controller configured to control the vehicle, sensors configured to detect objects in a path of the vehicle, nonvolatile memory including an artificial neural network configured to classify the objects detected by the sensors, and a processor. The artificial neural network includes a series of neurons in each of an input layer, at least one hidden layer, and an output layer. The memory includes instructions which, when executed by the processor, cause the processor to train the artificial neural network on a first task, identify, utilizing a contrastive excitation backpropagation algorithm, important neurons for the first task, identify, utilizing a learning algorithm, important synapses between the neurons for the first task based on the important neurons identified, and rigidify the important synapses to achieve selective plasticity of the series of neurons in the artificial neural network.