Neural Network Continual Learning via Architectural Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional deep learning neural networks face the challenge of 'catastrophic forgetting' when trained on multiple tasks, leading to deterioration in performance on previous tasks, making it impractical for resource-constrained platforms like mobile devices to handle increasing tasks effectively.
Innovation Solution
A novel training module that employs architectural search to identify optimal neural network structures for each task, allowing for sharing, adapting, or spawning new parameters, while minimizing loss function impact and controlling parameter size through regularizers, to reduce catastrophic forgetting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple neural networks are implemented to process multiple tasks, then task processing capability is improved, but device weight and bulk increase
Solution Approach 1:
The patent merges multiple task-specific neural networks into a single unified neural network that can process multiple tasks. The system uses a shared backbone network with task-specific adapter modules, allowing one network to perform multiple functions instead of requiring separate networks for each task, thereby reducing device weight and bulk.
Solution Approach 2:
The patent creates a universal neural network architecture that can handle multiple different tasks through a single model. The network uses parameter sharing and adaptive parameter mechanisms to universally process various tasks including image classification, object detection, and segmentation, eliminating the need for multiple specialized networks.
2Device complexity
If a single neural network is trained on multiple tasks, then device resource efficiency is improved, but catastrophic forgetting occurs leading to performance deterioration on previous tasks
Solution Approach 1:
The patent applies preliminary action by pre-training the shared backbone network on multiple tasks before fine-tuning. The system first learns general features across tasks, then adapts to specific tasks while preserving previously learned knowledge through careful parameter initialization and progressive training strategies.
Solution Approach 2:
The patent implements feedback mechanisms through continuous monitoring of performance on all previously learned tasks during training. The system uses loss functions that aggregate performance across multiple tasks and adjusts parameters to maintain balanced performance, preventing catastrophic forgetting through real-time feedback on overall system performance.
3Speed
If neural network parameters are directly modified for new tasks, then adaptation speed is improved, but performance on previous tasks deteriorates
Solution Approach 1:
The patent segments the neural network parameters into shared parameters and task-specific parameters. The shared parameters remain relatively stable and store general knowledge across tasks, while task-specific parameters are freely modified during adaptation. This segmentation allows rapid adaptation to new tasks without completely overwriting previously learned information.
Solution Approach 2:
The patent applies local quality by allowing different parts of the parameter space to have different degrees of plasticity. Some parameters are constrained to change slowly to preserve previous task performance, while others are allowed to change rapidly to adapt to new tasks. This localized parameter control enables selective adaptation without global performance deterioration.
Data Source
AI summary
Embodiments for training a neural network using sequential tasks are provided. A plurality of sequential tasks are received. For each task in the plurality of tasks a copy of the neural network that includes a plurality of layers is generated. From the copy of the neural network a task specific neural network is generated by performing an architectural search on the plurality of layers in the copy of the neural network. The architectural search identifies a plurality of candidate choices in the layers of the task specific neural network. Parameters in the task specific neural network that correspond to the plurality of candidate choices and that maximize architectural weights at each layer are identified. The parameters are retrained and merged with the neural network. The neural network trained on the plurality of sequential tasks is a trained neural network.