On-Device NLP Personalization with Adapters Against Catastrophic Forgetting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current continual learning systems suffer from catastrophic forgetting and inadequate knowledge transfer, particularly in natural language processing (NLP) models, making it challenging to adapt to new tasks or domains while preserving previous knowledge effectively.
Innovation Solution
A computer-implemented method using high-order pooling and adapter modules to personalize NLP models, which involves generating statistical descriptors from embedded representations and incorporating adapter modules to update the model efficiently on a user device, allowing it to adapt to new tasks and domains without retraining.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If continual learning is used to adapt NLP models to new tasks or domains, then adaptability is improved, but catastrophic forgetting occurs causing loss of previous knowledge
Solution Approach 1:
The model parameters are segmented into two distinct sets: basic parameters that remain fixed and are not updated during continual learning, and adapter parameters that are trained separately for each new task or domain. This segmentation allows the model to adapt to new tasks while preserving previously learned knowledge in the basic parameters, thereby resolving the catastrophic forgetting problem.
Solution Approach 2:
Adapter modules are introduced as intermediary components that sit between the pre-trained NLP model and the task-specific output layers. These adapters serve as mediators that enable task adaptation without directly modifying the core model parameters, thus preventing catastrophic forgetting while maintaining adaptability to new tasks and domains.
2Measurement precision
If full model retraining is performed to personalize NLP models, then accuracy is improved, but computational cost and time consumption increase significantly
Solution Approach 1:
The training process is extracted and focused only on the adapter parameters rather than the entire model. By taking out the adapter modules as separate trainable components, the system achieves task-specific personalization with minimal computational resources and time, while the majority of the model parameters remain frozen and do not require retraining.
Solution Approach 2:
Instead of performing complete model retraining (excessive action), the system applies partial training only to the adapter parameters. This partial action is sufficient to achieve task adaptation and personalization while dramatically reducing computational cost and training time compared to full model retraining.
3Adaptability or versatility
If adapter modules are added to enable task adaptation, then versatility is improved, but device complexity increases
Solution Approach 1:
The adapter modules are nested within the existing NLP model architecture, integrating seamlessly between the pre-trained model layers and the output layers. This nesting approach allows the system to gain task adaptation versatility while maintaining a relatively simple overall structure, as the adapters are contained within the existing framework rather than adding external complexity.
Data Source
AI summary
The present techniques generally relate to a computer-implemented method for using continual learning to personalise natural language processing (NLP) models to unseen tasks or domains. The models may be used on various downstream NLP applications, such as Text Classification (TC), Natural Language Inference (NLI), Document or Aspect Sentiment Classification (DSC or ASC). The framework or architecture which may be used as a natural language processing (NLP) model can be trained using continual learning. The framework employs three main modules. The first module is a tokeniser 100 which incorporates a set of adapter modules which allow for adaptation of the model to both new tasks and/or new domains. The second module is a reduction module 106 which uses high order embedding statistics (which may also be termed statistical descriptors) for modeling different characteristics of data from different domains and tasks. The third module is a classifier 108 in the form of a personalized multi-layer perceptron (MLP) head which is for modelling task-specific information and/or domain-specific information.