Neural Network Continual Learning via Channel-Wise Masks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing continual learning technologies face challenges such as catastrophic forgetting, especially when previous task data is not accessible due to privacy concerns, and they often require significant computational resources as the number of tasks increases.
Innovation Solution
A machine learning method and electronic device that utilize a neural network model and channel-wise masks to perform feature extraction and energy score conversion, allowing for self-adaptive identification of target tasks and preventing catastrophic forgetting by selectively updating neural network parameters based on loss functions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If joint learning is used to train all tasks simultaneously, then all tasks can be optimized together, but the training time is prolonged and data of new tasks are diluted by previous task data
Solution Approach 1:
The patent segments the training process into separate task-specific training phases instead of joint training. Each task is trained independently in sequence, with the model being fine-tuned on task-specific data without being diluted by other task data. This segmentation resolves the contradiction by maintaining high task optimization quality while reducing overall training time through parallelizable independent training stages.
2Reliability
If experience replay is used to provide previous task data to new tasks, then catastrophic forgetting is reduced, but previous task data must be stored which is not possible due to privacy concerns
Solution Approach 1:
The patent introduces task embedding vectors as an intermediary representation that captures essential task characteristics without requiring storage of actual task data. These embeddings serve as a compact summary of task knowledge that can be stored and reused, preventing catastrophic forgetting while respecting privacy constraints by not storing sensitive original data.
Solution Approach 2:
The patent transforms task-specific knowledge from raw data into parameter changes in the form of task embedding vectors and adapter parameters. Instead of storing previous task data, the system stores learned parameter adjustments that capture task-specific features, enabling knowledge retention without data storage requirements.
3Reliability
If dynamic network isolation is used to separate tasks, then catastrophic forgetting is prevented, but the number of parameters and data amount increase proportionally with tasks
Solution Approach 1:
The patent implements a universal base model that can handle multiple tasks through task-specific adapters and embeddings rather than creating separate isolated networks for each task. The shared base model maintains common knowledge while task-specific components handle task differentiation, reducing overall parameter count compared to fully isolated task networks while preventing catastrophic forgetting.
Solution Approach 2:
The patent employs a nested architecture where task-specific adapter modules are embedded within a shared base model. The task embeddings and adapters are nested within the universal model structure, allowing compact representation of multiple tasks within a unified framework rather than requiring separate standalone models for each task.
4Adaptability or versatility
If task-free continual learning is used, then practical application requirements are met, but the learning process relies on Softmax function confidence which causes overfitting in sample space
Solution Approach 1:
The patent introduces task embedding vectors as an intermediary between the input data and the Softmax classification layer. These embeddings provide task-specific contextual information that guides the Softmax function more accurately, reducing overfitting to sample space artifacts while maintaining practical task-free learning capabilities. The embeddings act as a mediator that prevents direct overfitting to training sample distributions.
Data Source
AI summary
A machine learning method for continual learning is provided, and the method includes following steps. Capturing an input image. Performing feature extraction on the input image by a plurality of sub-models to obtain a plurality of feature maps, where the sub-models correspond to a plurality of tasks, and the sub-models are determined by a neural network model and a plurality of channel-wise masks. Converting the feature maps into a plurality of energy scores. Selecting a target sub-model corresponding to a target task of the tasks from the sub-models according to the energy scores. Outputting a prediction result corresponding to the target task by the target sub-model.