Neural Network Weight Initialization for Continual Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Continual learning in convolutional neural networks (CNNs) faces the challenge of catastrophic forgetting, where accuracy on previous tasks decreases as new tasks are learned, limiting the number of tasks that can be added and the performance on target tasks.
Innovation Solution
A machine learning device and method that determines initialization rates for neural network weights based on layer depth and task similarity, allowing for efficient transfer of weights between tasks to maintain accuracy and increase the number of learnable tasks, using a machine learning device with initialization and inference units to generate and utilize initialized neural network models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If continual learning is performed in a CNN to learn new tasks, then the model can adapt to new tasks, but the accuracy on previous tasks decreases due to catastrophic forgetting
Solution Approach 1:
The patent segments the neural network weights into different groups based on their importance to previous tasks. By dividing weights into retained weights (important for previous tasks) and initialized weights (less important), the system can selectively update only the initialized weights for new tasks while preserving the segmented retained weights, thus preventing catastrophic forgetting
Solution Approach 2:
The patent performs preliminary action by determining importance scores for weights before learning new tasks. The importance determination unit calculates which weights are critical for previous tasks in advance, and the initialization unit pre-initializes less important weights before new task training begins. This preliminary classification ensures that when new tasks are learned, only non-critical weights are updated, protecting previous task performance
2Adaptability or versatility
If the entire dataset is retrained to adapt to changes in data distribution, then the model can adapt to new data, but the learning time and computational cost increase significantly
Solution Approach 1:
The patent extracts and identifies only the necessary weights that need to be updated for new tasks by calculating importance scores. Instead of retraining the entire model on the complete dataset, the system extracts and updates only the non-important weights that are less critical for previous tasks, dramatically reducing the amount of data and computation required for adaptation
Solution Approach 2:
The patent applies local quality by treating different weights differently based on their importance to previous tasks. Rather than uniformly updating all weights through full dataset retraining, the system applies gradient updates only to weights with lower importance scores, while protecting weights with high importance scores. This localized update approach reduces computational burden while maintaining adaptability
3Adaptability or versatility
If PackNet method is used for continual learning by iterative pruning, then multiple tasks can be added to a single network, but the number of additional tasks and accuracy of added tasks are not improved for the target task performance
Solution Approach 1:
The patent changes the parameter selection criterion from PackNet's fixed iterative pruning approach to a dynamic importance-based selection. Instead of uniformly pruning weights in fixed iterations, the system calculates importance scores for each weight based on its contribution to previous tasks and dynamically determines which weights to initialize. This parameter change enables more tasks to be learned while maintaining higher accuracy on target tasks
Data Source
AI summary
An initialization rate determination unit determines, in accordance with a depth of a layer in a neural network model, a first initialization rate for initializing weights in the neural network model on a first task. A machine learning execution unit generates a neural network model trained on a first task by training on the first task by machine learning. An initialization unit initializes weights in the neural network model trained on the first task, based on the first initialization rate, to generate an initialized neural network model trained on the first task, the initialized neural network trained on the first task being used in a second task.