Progressive Neural Networks With Lateral Transfer Across Tasks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network systems struggle with transfer learning across multiple tasks, often discarding previously learned knowledge and suffering from catastrophic forgetting, especially when tasks are unrelated or adversarial.
Innovation Solution
A progressive neural network system that integrates previously learned knowledge at each layer through lateral connections between deep neural networks, allowing for seamless integration of new tasks without interference, and naturally accumulating experiences.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a neural network is trained on a single task using conventional methods, then it achieves good performance on that task, but it forgets previously learned knowledge when new tasks are introduced
Solution Approach 1:
The neural network is segmented into multiple independent task-specific networks, each responsible for a specific task. This segmentation allows each task network to maintain its learned knowledge independently while being part of a larger progressive system, preventing catastrophic forgetting when new tasks are introduced.
Solution Approach 2:
The patent transitions from a single flat network architecture to a multi-dimensional progressive structure where networks are organized in a sequence across time/tasks. This dimensional transformation allows the system to preserve historical task representations while adding new capabilities, resolving the contradiction between learning new tasks and retaining old knowledge.
2Adaptability or versatility
If the neural network architecture is expanded to learn multiple tasks simultaneously, then task versatility improves, but training complexity and computational resources increase significantly
Solution Approach 1:
The system trains neural networks in a progressive sequence where each task is learned in advance before moving to the next. This preliminary action approach allows each task to be mastered independently with optimized resources, while the cumulative effect achieves multi-task versatility without requiring complex simultaneous training mechanisms.
Solution Approach 2:
Instead of creating a single complex multi-task network, the system creates simplified copies of the network architecture for each task. Each task-specific network is a copy trained on a specific task, and the collection of copies provides multi-task capability with reduced individual complexity and training requirements.
3Adaptability or versatility
If conventional neural networks are used for continual learning, then new tasks can be learned, but interference from previously learned tasks degrades performance
Solution Approach 1:
The neural network system is segmented into independent task-specific networks that process tasks separately. This segmentation isolates the representational spaces of different tasks, preventing interference between them while maintaining the ability to learn continually. Each task network preserves its performance stability independently.
Solution Approach 2:
The patent introduces task-specific adapter layers or projection mechanisms as intermediaries between the shared feature extraction components and task-specific output layers. These intermediaries allow different tasks to share common representations without direct interference, enabling continual learning while maintaining performance stability across all tasks.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods and systems for performing a sequence of machine learning tasks. One system includes a sequence of deep neural networks (DNNs), including: a first DNN corresponding to a first machine learning task, wherein the first DNN comprises a first plurality of indexed layers, and each layer in the first plurality of indexed layers is configured to receive a respective layer input and process the layer input to generate a respective layer output; and one or more subsequent DNNs corresponding to one or more respective machine learning tasks, wherein each subsequent DNN comprises a respective plurality of indexed layers, and each layer in a respective plurality of indexed layers with index greater than one receives input from a preceding layer of the respective subsequent DNN, and one or more preceding layers of respective preceding DNNs, wherein a preceding layer is a layer whose index is one less than the current index.