Class Prototype Initialization for Continual Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Classification models in continual learning settings face the challenge of 'catastrophic forgetting,' where they forget previously learned classes or data distributions when trained on new data, leading to decreased performance, especially in class-incremental settings, and existing methods require storing previous training data, which is resource-intensive.
Innovation Solution
A method for training classification models using a loss function with three components: one to learn sample representations, another to update initialized class prototypes by optimizing similarity with the model's output, and a third to preserve the order of relative distances between prototypes and model outputs, allowing for continual learning without storing previous data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a classification model is trained on new classes in continual learning, then the model learns new categories, but the classification performance on previously learned classes deteriorates due to catastrophic forgetting
Solution Approach 1:
The method performs preliminary actions by initializing class prototypes for new classes before training begins. These prototypes serve as pre-established reference points that guide the learning process, allowing the model to adapt to new classes while maintaining performance on previous classes through the preservation of prototype structures.
Solution Approach 2:
The method creates simplified copies of class characteristics through prototypes. Instead of storing all training data, the invention extracts and stores essential class features as prototypes, which can be updated during training to represent new classes while preserving information about previous classes, thereby preventing catastrophic forgetting.
2Reliability
If previous training data is stored and reused to prevent catastrophic forgetting, then classification performance on previous classes is maintained, but memory and computing resources are consumed
Solution Approach 1:
The method extracts only the essential characteristics of each class into compact prototype representations. By taking out only the most important features that define a class rather than storing complete training datasets, the invention maintains classification performance while dramatically reducing memory requirements.
Solution Approach 2:
The method changes the parameter representation from storing high-dimensional training samples to storing low-dimensional prototype vectors. This parameter transformation allows the model to preserve class information in a compressed form, reducing memory consumption while maintaining the ability to prevent catastrophic forgetting.
3Reliability
If previous training data is stored and reused to prevent catastrophic forgetting, then classification performance is maintained, but computing resources are consumed
Solution Approach 1:
The method extracts only the essential characteristics of each class into compact prototype representations. By taking out only the most important features that define a class rather than storing complete training datasets, the invention maintains classification performance while dramatically reducing memory requirements.
Solution Approach 2:
The method changes the parameter representation from storing high-dimensional training samples to storing low-dimensional prototype vectors. This parameter transformation allows the model to preserve class information in a compressed form, reducing memory consumption while maintaining the ability to prevent catastrophic forgetting.
4Device complexity
If class prototypes are learned using existing methods, then some class representation is achieved, but classification accuracy drops for inputs with classes absent in the current training stage
Solution Approach 1:
The method performs preliminary actions by initializing class prototypes for new classes before training begins. These prototypes serve as pre-established reference points that guide the learning process, allowing the model to adapt to new classes while maintaining performance on previous classes through the preservation of prototype structures.
Solution Approach 2:
The method implements feedback mechanisms where prototype updates are guided by loss functions that consider both new and previous class information. This feedback ensures that prototype evolution maintains classification accuracy across all classes, including those absent from the current training stage, by continuously adjusting prototypes based on their representational quality.
Data Source
Figure 1~2
Figure 3~4
AI summary
A method for training a classification model (f) with a training dataset (X,Y) and a set of class prototypes (pc), the method comprising an initialization of a class prototype for at least one class of the training dataset not having a class prototype in the set of class prototypes, and wherein training is performed using a loss function including: - a first component (LSC) to learn, by the classification model, representation of samples of the training dataset, - a second component (Lp) configured to update the at least one initialized class prototype by optimizing a similarity between the initialized class prototype and the output of the classification model, - a third component (Ld) configured to update at least one class prototype of the set of class prototypes and the classification model by preserving, during training, an order distances.