CNN Filter Sharing for Continual Learning Without Catastrophic Forgetting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Catastrophic forgetting occurs in convolutional neural networks (CNNs) due to the saturation of filters when learning new tasks, limiting the number of tasks that can be learned.
Innovation Solution
The method involves extracting overlap filters with high similarity in weight as shared filters and initializing the weights of other filters to 0, allowing new tasks to be learned while maintaining performance on previous tasks by using shared filters efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If filters are reused for new tasks in PackNet, then new tasks can be learned, but the number of learnable tasks is limited due to filter saturation
Solution Approach 1:
The patent segments filters into multiple groups and assigns different groups to different tasks. Instead of reusing the same filters for all tasks, the system divides the filter set so that each task can utilize a dedicated subset of filters, preventing saturation and enabling learning of more tasks simultaneously.
Solution Approach 2:
The patent makes filters multi-functional by allowing different filter groups to serve different tasks. Each filter group can be selectively activated for specific tasks, enabling the same overall filter set to handle multiple tasks without any single group becoming saturated.
2Adaptability or versatility
If CNN parameters are re-trained for new tasks, then new tasks can be learned, but precision on old tasks decreases due to catastrophic forgetting
Solution Approach 1:
The patent segments the parameter space by assigning different parameter groups to different tasks. When learning a new task, only the parameters assigned to that task are updated, while parameters for other tasks remain frozen, preventing catastrophic forgetting of previously learned tasks.
Solution Approach 2:
The patent applies local quality by allowing different parts of the network (different filter groups and their associated parameters) to have different update behaviors. Some parameters are updated for new tasks while others are kept fixed to preserve performance on old tasks.
Data Source
Figure 1A~1E
Figure 2
Figure 3A~3E
AI summary
A weight storage unit (50) stores weights of a plurality of filters used to detect a feature of a task. A continual learning unit (20) trains the weights of the filters in response to an input task in continual learning. A filter processing unit (30) locks, of a plurality of filters that have learned one task, the weights of a proportion of the filters to prevent the proportion of the filters from being used to learn a further task and initializes the weights of other filters to use the other filters to learn a further task. A comparison unit (40) compares the weights of a plurality of filters that have learned two or more tasks, extracts overlap filters having a similarity in weight over a threshold value as shared filters shared by tasks, leaves one of the overlap filters as the shared filter, and initializes the weights of filters other than the shared filter.