Teacher-Student Model Updating for Continual Learning Retention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large pre-trained models face challenges in continual learning scenarios due to catastrophic forgetting, leading to a decline in performance on previously learned tasks, especially in applications like autonomous driving, where forgetting previously learned information can compromise safety and efficiency.
Innovation Solution
A computer-implemented method for updating a teacher-student framework by selecting the most confident model based on a maximum logit score to generate a pseudo label, adjusting the student model's parameters to minimize prediction discrepancy, and updating the teacher model's parameters using an exponential moving average of the student's updated parameters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional continual learning methods (regularization, external memory, dynamic expansion) are applied to large pre-trained models, then the model can retain previously learned knowledge, but the model exhibits suboptimal performance even compared to zero-shot capabilities
Solution Approach 1:
The patent implements dynamic task selection during the updating process. The system dynamically determines which tasks should be retained and which new tasks should be learned by comparing performance metrics across different tasks, allowing the model to adaptively adjust its knowledge retention strategy rather than applying static regularization to all tasks equally
Solution Approach 2:
The patent changes the parameters of the pre-trained model selectively based on task importance and performance degradation metrics. Instead of freezing or uniformly updating all parameters, the system identifies specific parameter subsets that require updating for new tasks while preserving parameters critical for previously learned tasks, achieved through performance monitoring and selective parameter optimization
2Adaptability or versatility
If the model learns new information through continual learning, then the model adapts to new tasks, but catastrophic forgetting occurs and performance on previously learned tasks deteriorates
Solution Approach 1:
The patent implements a feedback mechanism where the model's performance on previously learned tasks is continuously monitored during the learning of new tasks. This performance feedback is used to adjust the updating process, allowing the system to detect when catastrophic forgetting is occurring and modify the learning strategy to prevent further degradation of previously learned capabilities
Solution Approach 2:
The patent performs preliminary evaluation of task importance and potential forgetting risks before updating the model with new information. By assessing which tasks are most vulnerable to forgetting and which parameters are critical for each task, the system prepares appropriate retention strategies in advance, preventing catastrophic forgetting before it occurs rather than reacting after performance degradation
Data Source
Figure 1~2
AI summary
A computer-implemented method (20) for updating a teacher-student framework comprising a teacher model (T) and a student model (S), the computer-implemented method (20) comprising selecting (21), for a given test sample, the most confident model (S, T) among the student model (S) and the teacher model (T) based on a maximum logit score to generate a pseudo label for said test sample; updating (22) initial parameters (θ) of the student model (S) based on the generated pseudo-label and updating initial parameters (θ') of the teacher model (T) based on the updated parameters (θn) of the student model (S).