Continual Few-Shot Learning Weight Generator Prevents Catastrophic Forgetting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional few-shot learning methods struggle to effectively utilize base classes during training, leading to a 'forgetting' issue where base classes are not preserved when novel classes are introduced, especially in limited-data scenarios.
Innovation Solution
The proposed method employs a weight generator that updates classification weights for novel classes based on both base and previously learned classification weights, using attention-based mechanisms and regularization to prevent forgetting of base classes, allowing for continual learning across multiple tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional few-shot learning trains a model with a small amount of training data without utilizing base classes, then the model can learn from limited data, but the model cannot utilize the knowledge from base classes to improve few-shot learning performance
Solution Approach 1:
The patent pre-trains a feature embedding network on base classes before performing few-shot learning. This preliminary action allows the model to acquire transferable knowledge from base classes that can be utilized when training data for novel classes is limited, thereby improving few-shot learning performance without significantly increasing training complexity during the actual few-shot task
Solution Approach 2:
The patent separates the training process into two distinct phases: (1) pre-training on base classes to learn transferable features, and (2) few-shot learning on novel classes using the pre-trained feature extractor. This segmentation allows the model to utilize base class knowledge while keeping the few-shot training simple and efficient
2Productivity
If the feature embedding network and classification weights for base classes are pre-trained and fixed, then the model can generate classification weights for novel classes, but the model forgets base classes when novel classes are introduced
Solution Approach 1:
The patent implements a feedback mechanism where the model periodically re-evaluates and re-trains on base classes even after learning novel classes. The loss function includes terms that ensure the model maintains its ability to classify base classes while learning novel classes, preventing catastrophic forgetting through continuous feedback from base class performance
Solution Approach 2:
The patent dynamically adjusts the classification weights for base and novel classes using a unified loss function that balances performance on both types of classes. By changing the parameters (classification weights) in a controlled manner that considers both base and novel class performance, the model avoids forgetting base classes while adapting to novel classes
3Adaptability or versatility
If classification weights for novel classes are trained using only samples of novel classes, then the model can learn novel classes, but the model experiences forgetting issue for base classes
Solution Approach 1:
The patent merges the training objectives for base classes and novel classes into a unified loss function. This combined loss function simultaneously optimizes performance on both base and novel classes, ensuring that the model learns novel classes while maintaining reliability on base classes through balanced gradient updates from both data sources
Data Source
AI summary
Methods and apparatuses are provided for continual few-shot learning. A model for a base task is generated with base classification weights for base classes of the base task. A series of novel tasks is sequentially received. Upon receiving each novel task in the series of novel tasks, the model is updated with novel classification weights for novel classes of the respective novel task. The novel classification weights are generated by a weight generator based on one or more of the base classification weights and, when one or more other novel tasks in the series are previously received, one or more other novel classification weights for novel classes of the one or more other novel tasks. Additionally, for each novel task, a first set of samples of the respective novel task are classified into the novel classes using the updated model.