Apparatus for performing class incremental learning and method of operating the apparatus
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks face the catastrophic forgetting phenomenon when retrained, leading to loss of existing learning information as new classes are added, which is a challenge in class incremental learning.
Innovation Solution
The method involves training an autoencoder using first and second input embeddings for different class groups, calculating contribution values and representative values, and updating these values during retraining, with loss functions incorporating cosine similarity and regularization terms to prevent forgetting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the neural network is retrained with new classes, then the network can recognize new classes, but the network loses previously learned information (catastrophic forgetting)
Solution Approach 1:
The patent calculates contribution values of parameters before retraining occurs, and uses these pre-calculated values to guide the retraining process. By preparing the contribution values in advance, the system can prioritize which parameters to update and which to preserve, thereby preventing catastrophic forgetting while still acquiring new class information
Solution Approach 2:
The patent implements a feedback mechanism where contribution values are calculated based on the importance of parameters for existing classes, and this feedback is used to adjust the retraining process. The loss function incorporates these contribution values to provide feedback during parameter updates, ensuring that parameters critical for existing classes are protected while allowing updates for new classes
2Adaptability or versatility
If all parameters are updated during retraining, then the network adapts to new classes, but the accuracy on existing classes deteriorates
Solution Approach 1:
The patent applies different update strategies to different parameters based on their contribution values. Parameters with high contribution values (critical for existing classes) are protected from updates or updated with smaller learning rates, while parameters with low contribution values are freely updated. This local differentiation of update quality prevents accuracy deterioration on existing classes while enabling adaptation to new classes
Solution Approach 2:
The patent dynamically changes parameter update rules based on calculated contribution values. The loss function is modified to include regularization terms weighted by contribution values, effectively changing the optimization landscape to protect important parameters while allowing flexibility for less important ones. This parameter-specific adaptation resolves the contradiction between updating for new classes and maintaining accuracy for existing classes
Data Source
AI summary
A method of performing a class incremental learning in a neural network apparatus, the method including training an autoencoder using first input embeddings with respect to a first class group, calculating a contribution value of each of parameters of the autoencoder and calculating a representative value with respect to each of at least one first class included in the first class group in the training of the autoencoder, retraining the autoencoder using second input embeddings with respect to a second class group, and updating the contribution value of the each of the parameters and calculating a representative value with respect to each of at least one second class included in the second class group in the retraining the autoencoder.