Continual Learning Model Reset via Knowledge Distillation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Continual reinforcement learning (CRL) faces challenges with catastrophic forgetting (CF) and negative transfer, where previously learned knowledge is forgotten and interfered with due to intrusive information from previous tasks, leading to decreased accuracy and learning failures.
Innovation Solution
The method involves learning a first model based on a reinforcement learning algorithm, performing knowledge distillation from the first model to a second model, and using behavioral cloning on the second model, while storing and updating information in buffers to reset the first model and prevent CF and negative transfer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a reinforcement learning model continuously learns from new tasks, then the model adapts to new data, but previously learned knowledge is forgotten due to catastrophic forgetting
Solution Approach 1:
The method performs preliminary actions by storing information from previous tasks in a buffer before encountering catastrophic forgetting. The buffer preserves historical task information, and when forgetting occurs, the system can retrieve and relearn from these stored representations, preventing complete loss of previously learned knowledge.
Solution Approach 2:
The system creates copies of task information by generating latent representations and storing them in buffers. Instead of maintaining the original full task data, the system copies essential task characteristics into compressed latent forms that can be retrieved and used for recovery when catastrophic forgetting occurs.
2Adaptability or versatility
If the model learns from multiple tasks sequentially, then the model gains versatility across tasks, but accuracy decreases due to negative transfer from intrusive information
Solution Approach 1:
The method segments task information by creating separate latent representations for different tasks and storing them in organized buffers. This segmentation prevents intrusive information from one task from contaminating another task's learning, as each task maintains its own distinct representation that can be selectively activated without negative transfer.
Solution Approach 2:
The latent space acts as an intermediary between multiple tasks. Instead of tasks directly interfering with each other in the model weights, the system mediates through latent representations that capture task-specific information in a controlled manner, reducing negative transfer effects while maintaining versatility.
Data Source
AI summary
A method and device for performing continual learning are provided. The method of performing continual learning of tasks in a set of tasks includes learning a first model based on training data corresponding to a current task in the set of tasks, learning a second model based on information on the current task and information on a previous learning task in the set of tasks, and resetting the first model.