Adaptive Learning Rate Continual Learning for Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing continual learning methods face challenges in preventing catastrophic forgetting, where performance on previous training data deteriorates when new data is introduced, and they lack quantitative measures to assess convergence and performance changes.
Innovation Solution
A memory-based continual learning algorithm that computes adaptive learning rates for both previous and new training data using gradients, allowing the neural network to maintain performance on previous data while accommodating new data, and provides indicators to quantify learning performance such as bias and forgetting degree.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional continual learning methods are used to learn new data, then new data performance is improved, but performance on previous training data deteriorates due to catastrophic forgetting
Solution Approach 1:
The patent stores a subset of previous training data in memory before new data arrives. This preliminary preservation of historical data enables the model to periodically review and reinforce previous knowledge, preventing catastrophic forgetting while learning new tasks. The memory buffer acts as a safeguard that maintains access to historical information needed for continual learning.
Solution Approach 2:
The patent dynamically adjusts the learning rate parameter based on whether the current batch contains previous or new training data. By changing the learning rate parameter adaptively - using lower rates for previous data to preserve learned patterns and higher rates for new data to accelerate learning - the system resolves the contradiction between maintaining historical performance and adapting to new tasks.
2Productivity
If gradient descent is used for continual learning, then learning efficiency is improved, but theoretical guarantee of convergence and quantitative measurement of performance change are lost
Solution Approach 1:
The patent introduces feedback mechanisms that compute and utilize the inner product between gradients of previous and new training data. This feedback provides quantitative information about the relationship between old and new learning tasks, enabling the system to measure performance changes and adjust learning strategies accordingly. The feedback loop restores theoretical convergence guarantees while maintaining gradient descent efficiency.
Solution Approach 2:
The patent replaces the conventional gradient descent mechanism with an adaptive variant that incorporates theoretical convergence analysis. By substituting the basic gradient descent with a theoretically-grounded adaptive method that uses gradient inner products and biased gradient estimators, the system maintains computational efficiency while regaining mathematical guarantees of convergence and performance measurement.
3Device complexity
If a fixed learning rate is used for training, then training process is simplified, but adaptive learning rates cannot be set for previous and new training data respectively
Solution Approach 1:
The patent transforms the static learning rate into a dynamic parameter that changes based on the type of training data being processed. The learning rate becomes adaptive, automatically adjusting its value depending on whether the current batch consists of previous or new training data. This dynamic adjustment enables differentiated learning strategies without requiring complex manual configuration.
Solution Approach 2:
The training system automatically determines and adjusts the appropriate learning rate based on the data type being processed, without requiring external intervention or complex configuration. The system serves itself by computing gradient inner products and autonomously selecting learning rate values, simplifying the overall process while achieving adaptive learning rate adjustment for different data types.
Data Source
AI summary
According to an aspect of the present invention, there is provided a method of training an artificial neural network based on a memory-based continual learning algorithm, which is performed by a computing device including at least one processor. The method includes: storing part of previous training data, used in previous training, in memory; for an artificial neural network trained using the previous training data, computing adaptive learning rates to be applied to the artificial neural network based on a first gradient for first batch data sampled from the memory and a second gradient for second batch data including part of new training data; and training the artificial neural network based on the first batch data, the second batch data, and the adaptive learning rates.