CNN Filter Sharing to Mitigate Continual Learning Saturation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Catastrophic forgetting in convolutional neural networks (CNNs) occurs when new tasks are learned, leading to a saturation of filters and a limited number of tasks that can be learned, as existing solutions like PackNet face filter saturation issues.
Innovation Solution
The proposed solution involves extracting overlap filters with high similarity in weight as shared filters, initializing other filters to 0, and excluding them from training to mitigate filter saturation, thereby increasing the number of tasks that can be learned.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If PackNet is used to avoid catastrophic forgetting, then old tasks are preserved, but the number of filters becomes saturated and the number of learnable tasks is limited
Solution Approach 1:
The patent segments filters into three distinct groups: shared filters (for old tasks), new filters (for current task), and discarded filters. This segmentation allows the network to maintain multiple task capabilities simultaneously by allocating different filter groups to different task requirements, thereby resolving the contradiction between preserving old tasks and learning new tasks.
Solution Approach 2:
The patent introduces a temporal dimension to filter management by maintaining filters across different time steps (old tasks vs. current task). Instead of sequentially replacing filters, the system operates in multiple temporal dimensions simultaneously, keeping old filters intact while adding new filters for current tasks, thus expanding the capacity for learnable tasks without sacrificing old task performance.
2Productivity
If filters are reused across multiple tasks, then resource utilization improves, but filter saturation occurs limiting further task learning
Solution Approach 1:
The patent applies local quality by assigning different functional roles to different groups of filters within the same network. Shared filters are optimized for old tasks, while new filters are dedicated to current tasks. This localized functional differentiation allows efficient resource utilization within each filter group while maintaining overall system adaptability for learning multiple tasks.
Solution Approach 2:
The patent dynamically changes the parameter of filter availability by controlling which filters are active for which tasks. Through the filter management unit, the system adjusts filter parameters (active/inactive state) based on task requirements, allowing the same physical filters to serve different functional purposes at different times, thereby increasing both resource utilization and task capacity.
Data Source
Figure 1A~1E
Figure 2
Figure 3A~3E
AI summary
A weight storage unit (50) stores weights of a plurality of filters used to detect a feature of a task. A continual learning unit (20) trains the weights of the plurality of filters in response to an input task in continual learning. A filter control unit (40) compares, after a predetermined epoch number has been learned in continual learning, the weight of a filter that has learned the task with the weight of a filter that is learning the task, extracts overlap filters having a similarity in weight equal to or greater than a predetermined threshold value as shared filters shared by tasks, and leaves one of the overlap filters as the shared filter and initializes the weights of filters other than the shared filter.