Dynamic Prototype Self-Attention for Rehearsal-Free Continual Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing continual learning methods suffer from performance deterioration due to small buffer sizes and data privacy concerns, and lack effective utilization of backbone features, leading to excessive overwriting and forgetting of previous knowledge.
Innovation Solution
A method for rehearsal-free continual learning using dynamic prototypes and self-attention, which computes class-wise prototypes after each task, integrates them with a PSAB module, and applies regularization and Gini losses to optimize parameter updates, ensuring efficient knowledge retention and adaptation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a rehearsal buffer is used to store past data for experience replay, then the model can retain previous knowledge, but storage requirements increase and data privacy concerns arise
Solution Approach 1:
The patent extracts only the essential features of past data by computing class-wise prototypes (mean feature representations) instead of storing entire data samples. This extraction approach retains the core knowledge needed for previous tasks while dramatically reducing storage requirements from storing complete datasets to storing only compact prototype vectors.
Solution Approach 2:
The patent transforms the storage representation from raw data samples to compressed prototype parameters (mean feature vectors). By changing the parameter representation from full data to condensed statistical summaries, the system maintains knowledge retention capability while minimizing storage footprint and eliminating privacy risks associated with storing actual data samples.
2Adaptability or versatility
If existing prompting-based methods are used without backbone feature utilization, then the model can adapt to new tasks, but previous knowledge is excessively overwritten or forgotten
Solution Approach 1:
The patent merges the utilization of backbone features with prompting-based continual learning by integrating prototype information (extracted from backbone features of previous tasks) into the prompt pool. This combination allows the model to leverage both the adaptability of prompting mechanisms and the knowledge retention capability of backbone feature utilization, preventing excessive forgetting of previous tasks.
Solution Approach 2:
The patent performs preliminary computation of class-wise prototypes from backbone features after each task training stage. By pre-computing and storing these prototype representations before new tasks arrive, the system prepares knowledge retention mechanisms in advance, ensuring that when new tasks are learned, the model has preserved essential information from previous tasks through the prompt pool.
3Quantity of substance
If rehearsal-based methods are used with small buffer sizes, then storage requirements are reduced, but performance deterioration occurs
Solution Approach 1:
The patent creates a compressed copy of past knowledge in the form of prototype vectors (mean feature representations) and stores these in the prompt pool. This copying approach captures the essential characteristics of previous tasks in a compact form, enabling effective knowledge retention with minimal storage requirements and avoiding the performance deterioration associated with small rehearsal buffers.
Data Source
AI summary
A method and apparatus for prompting-based rehearsal-free continual learning with dynamic prototypes-involved self-attention. A machine learning model with a base training set accepts continual new data inputs and use them to compute a prototype self-attention block which updates the model for the task associated with the new data without excessive or undesired overwriting of the existing model. Gini penalties are applied to the results of such computation to prevent excessive or undesired overwriting of the existing model.