Teacher-Student Signal Learning to Prevent Catastrophic Forgetting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Catastrophic forgetting occurs when a learned model is transferred from a source domain to an application domain with different environments, leading to significant deterioration in estimation accuracy, and existing methods that associate signal labels with learning data heavily burden human effort.
Innovation Solution
A learning device that extracts clean and mixture feature values using neural networks, calculates a value based on teacher and student vector representations, and learns the student model to align with the source domain teacher model without using labels, thereby preventing catastrophic forgetting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a learned model obtained by performing learning in the source domain is relearned in the application domain, then the model adapts to the new environment, but estimation accuracy deteriorates significantly due to catastrophic forgetting
Solution Approach 1:
The patent applies partial fine-tuning by selectively updating only certain parameters or layers of the neural network during relearning in the application domain, rather than updating all parameters. This partial action allows the model to adapt to the new environment while preserving previously learned knowledge from the source domain, thus preventing catastrophic forgetting and maintaining estimation accuracy.
Solution Approach 2:
The patent implements different update strategies for different parts of the neural network. Some layers or parameters are fully updated to adapt to the application domain, while other layers are preserved or updated minimally to retain source domain knowledge. This local differentiation allows simultaneous adaptation and knowledge preservation.
2Measurement precision
If supervised learning is performed by associating signal labels with learning data, then learning accuracy improves, but human effort and workload increase significantly
Solution Approach 1:
The patent enables the system to perform self-supervised learning where the model learns from unlabeled data by utilizing internal consistency, reconstruction objectives, or self-generated labels. This eliminates the need for manual label association while maintaining learning effectiveness, thus reducing human effort and automating the process.
Solution Approach 2:
The patent introduces an intermediate representation or auxiliary task that bridges the gap between unlabeled data and learning objectives. By using this intermediary, the model can learn meaningful patterns without requiring explicit human-labeled data, thus maintaining learning accuracy while reducing manual workload.
Data Source
AI summary
A learning device includes an acquisition unit that acquires a clean signal, a mixture signal and a source domain teacher learned model, an extraction unit that extracts a clean feature value by using the clean signal, an estimation unit that estimates a teacher vector representation by using the source domain teacher learned model and the clean feature value, an extraction unit that extracts a mixture feature value by using the mixture signal, an estimation unit that estimates a student vector representation by using a student learning model and the mixture feature value, a calculation unit that calculates a value based on the teacher vector representation and the student vector representation, and a learning unit that learns the student learning model by using the value so that estimation by the student learning model becomes closer to estimation by the source domain teacher learned model.