Cross-Connected Neural Paths Prevent Catastrophic Forgetting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning models face the challenge of catastrophic forgetting, where old knowledge is lost when learning new tasks, especially in class-incremental scenarios, due to the lack of effective methods to preserve previous knowledge.
Innovation Solution
The proposed solution involves the Disjointing and Sharing Past Net (DSPN) model, which creates distinct paths for each task and adds cross connections between tasks to share complementary information, thereby preventing catastrophic forgetting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a deep learning model learns new tasks sequentially, then the model's ability to perform new tasks is improved, but the model loses previous knowledge (catastrophic forgetting)
Solution Approach 1:
The neural network is segmented into multiple distinct paths, where each path is dedicated to a specific task. This segmentation allows the model to learn new tasks through new paths without interfering with previously learned tasks, thereby preventing catastrophic forgetting while maintaining adaptability to new tasks.
Solution Approach 2:
Cross-connection layers are introduced as intermediary components between distinct task paths. These cross-connections enable knowledge transfer and sharing between tasks, allowing the model to leverage previously learned knowledge when learning new tasks without directly modifying the original task paths, thus preventing catastrophic forgetting.
2Reliability
If distinct paths are created for each task, then catastrophic forgetting is prevented, but the model complexity increases
Solution Approach 1:
The cross-connection layers serve multiple functions: they enable knowledge transfer between tasks, facilitate information sharing across distinct paths, and provide a mechanism for preventing catastrophic forgetting. This multi-functionality reduces the need for separate complex mechanisms for each objective, thereby managing overall model complexity while maintaining knowledge preservation.
3Loss of information
If cross connections are added between tasks, then knowledge sharing is improved, but the computational overhead increases
Solution Approach 1:
Cross-connections are selectively applied between specific task paths based on their relevance and compatibility. Rather than creating dense connections between all tasks, the model establishes localized cross-connections where they are most beneficial, reducing unnecessary computational overhead while maintaining effective knowledge sharing where needed.
Data Source
AI summary
A method, a computer system, and a computer program product for using distinct paths with cross connections for distinct tasks to prevent catastrophic forgetting in class-incremental scenarios. Embodiments of the present invention may include receiving one or more tasks sequentially. Embodiments of the present invention may include applying one or more shareable blocks to the one or more tasks. Embodiments of the present invention may include learning one or more distinct paths for the one or more tasks. Embodiments of the present invention may include adding one or more cross connections between the one or more tasks. Embodiments of the present invention may include adding an aggregation block to collect one or more outputs from the distinct paths of each of the one or more tasks. Embodiments of the present invention may include providing a prediction.