Named Entity Recognition Incremental Learning Without Catastrophic Forgetting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep learning models face challenges in adding new classes without interfering with the classification of original classes, often requiring large datasets for re-annotation and are prone to catastrophic forgetting, which is time-consuming and resource-intensive.
Innovation Solution
A method for few-shot incremental learning that saves last hidden state features from a pre-trained model and uses cosine similarity and cross-entropy loss optimization to add new classes, avoiding catastrophic forgetting and reducing the need for large datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a deep learning model is retrained with all original training data enhanced with new classes, then the model can learn new classes, but the process is time-consuming and resource-consuming
Solution Approach 1:
The patent extracts and saves the last hidden state features corresponding to prototypes of original classes from the pre-trained model. These extracted features serve as a compact representation of original class knowledge, allowing the model to learn new classes without reprocessing all original training data, thus significantly reducing training time while maintaining adaptability.
Solution Approach 2:
The patent performs preliminary action by saving the hidden state features of original classes before adding new classes. This pre-computed feature representation is stored and reused during incremental learning, eliminating the need to retrain on original data when new classes are introduced, thereby reducing time consumption.
2Adaptability or versatility
If a deep learning model is retrained with all original training data enhanced with new classes, then the model can learn new classes, but the process is resource-consuming
Solution Approach 1:
The patent extracts and saves the last hidden state features corresponding to prototypes of original classes from the pre-trained model. These extracted features serve as a compact representation of original class knowledge, allowing the model to learn new classes without reprocessing all original training data, thus significantly reducing training time while maintaining adaptability.
Solution Approach 2:
The patent performs preliminary action by saving the hidden state features of original classes before adding new classes. This pre-computed feature representation is stored and reused during incremental learning, eliminating the need to retrain on original data when new classes are introduced, thereby reducing time consumption.
3Adaptability or versatility
If a deep learning model is trained to add new classes, then the model gains new classification capabilities, but classification accuracy for original classes deteriorates
Solution Approach 1:
The patent introduces an intermediary mechanism by using saved hidden state features of original classes as a reference during incremental learning. These saved features act as a mediator that guides the model to maintain accurate classification of original classes while learning new classes, preventing catastrophic forgetting through feature-level knowledge preservation.
Solution Approach 2:
The patent applies parameter changes by optimizing the loss function to include both cross-entropy loss for new classes and a constraint term that measures the distance between current hidden state features and saved hidden state features of original classes. This parameter adjustment ensures that while learning new classes, the model's representation of original classes remains stable, maintaining classification accuracy.
4Quantity of substance
If few shot learning is used to train with minimal examples, then annotation costs are reduced, but the model underperforms compared to a model trained with a large dataset
Solution Approach 1:
The patent performs preliminary action by pre-training the model on a large dataset to learn robust feature representations. The hidden state features from this pre-trained model are then saved and used as a foundation for few-shot incremental learning. This preliminary training endows the model with strong general capabilities, allowing it to achieve good performance even with minimal additional training data for new classes.
Solution Approach 2:
The patent applies parameter changes by optimizing the loss function to include both cross-entropy loss for new classes and a constraint term that measures the distance between current hidden state features and saved hidden state features of original classes. This parameter adjustment ensures that while learning new classes, the model's representation of original classes remains stable, maintaining classification accuracy.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
Enables accurate classification of both old and new classes with minimal training data, reducing annotation costs and resource consumption while maintaining model flexibility.
Implementation Method 1
The cosine similarity loss optimization is of last hidden state features resulting from the further trained deep learning model and the first class and the second class compared to the saved last hidden state features
Implementation Method 2
The cross entropy loss optimization is for classification of the first class, the second class, and the third class
Data Source
AI summary
A method, computer system, and a computer program product for classification training are provided. A deep learning model is trained with a first dataset that includes annotated samples that include a first and second class. Last hidden state features corresponding to respective tokens from prototypes of the first and second class are saved. The trained deep learning model is further trained with a second dataset that includes additional annotated samples that include the first and second class and a third class. The further training includes performing a cosine similarity loss optimization and a cross entropy loss optimization. The cosine similarity loss optimization is of last hidden state features resulting from the further trained deep learning model and the first class and the second class compared to the saved last hidden state features. The cross entropy loss optimization is for classification of the first, second, and the third class.