Neural Network Neuron Partitioning to Limit Catastrophic Forgetting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models face catastrophic forgetting when trained on new tasks, leading to a loss of previously learned capabilities, which hinders their robustness and training efficiency.
Innovation Solution
A task management system that partitions neurons into redundant and non-redundant sets within a neural network, fine-tunes non-redundant neurons, and trains redundant neurons independently to minimize forgetting, allowing the network to learn multiple tasks efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a machine learning model is trained on a new task, then the model's capability to perform the new task is improved, but the model's capability to perform previously learned tasks deteriorates due to catastrophic forgetting
Solution Approach 1:
The patent segments the neural network into two distinct parts: a frozen backbone network that maintains previously learned tasks and a trainable task-specific network that learns new tasks. This segmentation allows the model to improve adaptability to new tasks while preserving reliability on previous tasks, as the backbone remains unchanged and the task-specific network operates independently.
Solution Approach 2:
The patent extracts the task-specific learning functionality into a separate task-specific network that is trained independently of the backbone. By taking out the new task learning into a separate module, the system avoids catastrophic forgetting while improving new task capability, as the extracted module can be trained without affecting the backbone's previously learned representations.
2Adaptability or versatility
If a machine learning model is trained sequentially on multiple tasks, then the model's adaptability to different tasks is improved, but the training time and computational resources required increase
Solution Approach 1:
The patent segments the training process into two independent parts: training the backbone network once for previously learned tasks, and then training only the task-specific network for new tasks. This segmentation dramatically reduces training time compared to retraining the entire model, as only the task-specific network parameters are updated for each new task.
Solution Approach 2:
The patent performs preliminary action by pre-training the backbone network with previously learned tasks before introducing new tasks. This preliminary training establishes a solid foundation that can be reused for subsequent tasks, eliminating the need to retrain the entire model and significantly reducing overall training time.
3Adaptability or versatility
If a machine learning model is trained on more diverse tasks, then the model's versatility is improved, but the computational resources and training data requirements increase
Solution Approach 1:
The patent segments the model into a shared backbone and task-specific networks, allowing the same training data and computational resources to support multiple tasks. The backbone is trained once with diverse task representations, and then the task-specific networks can be trained with smaller, task-specific datasets, reducing overall resource requirements while maintaining versatility.
Solution Approach 2:
The patent creates a universal backbone network that serves multiple functions across different tasks. This universal backbone can be trained once with diverse task data and then reused for training task-specific networks, reducing the total quantity of training data and computational resources needed compared to training separate models for each task.
Data Source
AI summary
Techniques are described herein for a method of determining a similarity of each neuron in a layer of neurons of a neural network model to each other neuron in the layer of neurons. The method further includes determining a redundant set of neurons and a non-redundant set of neurons based on the similarity of each neuron in the layer. The method further includes fine tuning the set of non-redundant neurons using a first set of training data. The method further includes training the set of redundant neurons using a second set of training data.