Redundant Neuron Partitioning in Multi-Task Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models face catastrophic forgetting when trained on new tasks, leading to a loss of previously learned capabilities and hindering robustness and training efficiency.
Innovation Solution
A task management system partitions neurons into redundant and non-redundant sets within a neural network, fine-tunes non-redundant neurons, and perturbs redundant neurons to minimize forgetting, allowing sequential learning of multiple tasks without degrading previous performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a machine learning model is trained on a new task, then the model's capability to perform the new task is improved, but the model's capability to perform previously learned tasks deteriorates due to catastrophic forgetting
Solution Approach 1:
The patent segments the neural network into redundant neurons and non-redundant neurons based on their importance for previously learned tasks. Non-redundant neurons are those critical for maintaining accuracy on old tasks, while redundant neurons can be repurposed for new tasks without compromising old task performance. This segmentation allows selective training that prevents catastrophic forgetting.
Solution Approach 2:
The patent applies different training strategies to different parts of the neural network. Non-redundant neurons are protected from updates that would degrade old task performance, while redundant neurons are freely updated for new tasks. This local differentiation of training quality enables simultaneous maintenance of old skills and acquisition of new skills.
2Productivity
If redundant neurons are trained to perform a new task, then training efficiency is improved, but the accuracy of non-redundant neurons performing the first task may be affected
Solution Approach 1:
The patent extracts the training responsibility for new tasks from the entire neural network and assigns it specifically to redundant neurons. By taking out the new task training from the global network update process and isolating it to specific neuron subsets, the system prevents training noise from propagating to non-redundant neurons, thus preserving their accuracy.
Solution Approach 2:
Redundant neurons act as intermediaries that absorb the training burden for new tasks. These neurons serve as a buffer layer that can be updated without directly affecting the critical non-redundant neurons, thereby mediating between the need for new task learning and the need to preserve old task accuracy.
Data Source
AI summary
Techniques are described herein for a method of determining a similarity of each neuron in a layer of neurons of a neural network model to each other neuron in the layer of neurons. The method further includes determining a redundant set of neurons and a non-redundant set of neurons based on the similarity of each neuron in the layer. The method further includes fine tuning the set of non-redundant neurons using a first set of training data. The method further includes training the set of redundant neurons using a second set of training data.