Neural Network Catastrophic Forgetting Mitigation via Pseudo-Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Artificial neural networks (ANNs) face the challenge of catastrophic forgetting, where they tend to forget previously learned information when adapting to new data, making it difficult to build upon existing models while conserving previous mapping functions, especially in scenarios where training data distribution changes over time.
Innovation Solution
The method involves training an initial ANN with pseudo data generated by another ANN, transferring knowledge between ANNs, and using auto-associative functions to replicate input samples, generating pseudo data that helps maintain old memories during new learning stages, allowing for incremental learning without losing previously learned information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If an ANN is trained with new information to improve adaptability, then the ANN can learn new data distributions, but the ANN forgets previously learned information (catastrophic forgetting)
Solution Approach 1:
The patent applies preliminary action by generating pseudo-data from the ANN's current state before training with new information. This pseudo-data serves as a preliminary representation of previously learned knowledge, allowing the network to retain important patterns while adapting to new data distributions. The pseudo-data generation occurs in advance of the forgetting problem manifesting.
Solution Approach 2:
The patent uses copying by creating pseudo-data that replicates the essential characteristics of previously learned information. Instead of storing raw training data, the system copies the learned representations through pseudo-data generation, which can then be used during subsequent training to prevent catastrophic forgetting while requiring minimal memory resources.
2Loss of information
If historic training data is stored in a buffer to prevent forgetting, then previous knowledge is retained, but memory resources are consumed
Solution Approach 1:
The patent replaces physical storage of historic training data with generated pseudo-data copies. This approach maintains the essential information needed to prevent catastrophic forgetting while consuming minimal memory resources, as the pseudo-data is generated on-demand from the network's current parameters rather than storing extensive historical datasets.
Solution Approach 2:
The patent transforms the problem from storing data to storing parameters. By generating pseudo-data from the network's learned parameters rather than storing original training data, the system achieves knowledge retention with minimal memory footprint, effectively changing the state representation from data-centric to parameter-centric.
3Stability of the object's composition
If the ANN architecture and parameters are fixed to maintain stability, then previously learned mapping functions are preserved, but the ANN cannot adapt to new data distributions
Solution Approach 1:
The patent applies preliminary action by generating pseudo-data that captures the essential characteristics of previously learned mappings before training with new data. This preliminary pseudo-data serves as a stabilizing influence during subsequent adaptation, allowing the network to maintain its learned mapping functions while adapting to new data distributions.
Solution Approach 2:
The patent introduces pseudo-data as an intermediary between the network's existing knowledge and new training data. This intermediary element facilitates the transition by providing a bridge that connects previously learned patterns with new information, enabling stable adaptation without catastrophic forgetting.
Data Source
AI summary
A method of training an artificial neural network, the method comprising: initially training a first artificial neural network with first input data and first pseudo data, wherein the first pseudo data is or was generated by a second artificial neural network in a virgin state, or by the first artificial neural network while in a virgin state; generating second pseudo data using the first artificial neural network, or using the second artificial neural network following at least partially transferring knowledge from the first artificial neural network to the second artificial neural network; and training the first artificial neural network, or another artificial neural network, with the second pseudo data and second input data.