LSTM Hidden State Sharing to Prevent Catastrophic Forgetting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Artificial neural networks face the challenge of catastrophic forgetting, where they abruptly forget previously learned information when learning new tasks, making it difficult to retain useful knowledge and skills from old tasks without requiring task identifiers during the inference stage, which is impractical in real-world scenarios.
Innovation Solution
Utilizing hidden state sharing modules in a long short term memory structure to broadcast and collect hidden states across different tasks, allowing information sharing and preventing the loss of previously learned information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If an artificial neural network learns new tasks, then the network's adaptability improves, but previously learned information is lost due to catastrophic forgetting
Solution Approach 1:
The patent segments the neural network into multiple task-specific modules, each responsible for a particular task. This segmentation allows the network to maintain separate knowledge representations for different tasks, preventing catastrophic forgetting when learning new tasks while still improving adaptability through modular task handling.
Solution Approach 2:
The patent introduces a task identifier as an intermediary input that routes information through appropriate task-specific modules. This intermediary mechanism enables the network to selectively access and retain relevant information from different tasks, preventing information loss while maintaining adaptability to new tasks.
2Loss of information
If task identifiers are used during the inference stage to prevent catastrophic forgetting, then information retention improves, but the system complexity and operational requirements increase
Solution Approach 1:
The patent performs preliminary action by training the network to automatically generate or infer task identifiers during the training phase. This preliminary learning of task identification eliminates the need for external task identifier inputs during inference, reducing system complexity while maintaining information retention capabilities.
Solution Approach 2:
The network is designed to self-determine the appropriate task context without external intervention. By learning to automatically identify and select relevant task modules based on input characteristics, the system serves itself, eliminating the need for manual task identifier provision and reducing operational complexity.
Data Source
AI summary
A computer-implemented method, system and computer program product for processing data. Data, including single data points (e.g., images) or entire sequences of data (e.g., speech, video), is received to be processed. A long short term memory structure is utilized to process the received data, where the long short term memory structure includes hidden state sharing modules for allowing information sharing in hidden states across different tasks. The hidden state sharing modules include broadcast modules which are configured to send hidden states of the current task to all previous modules and collect modules which are configured to collect all the hidden states from all the previous modules. In this manner, catastrophic forgetting is avoided by preventing the loss of previously learned information via the use of hidden state sharing modules.