Neural Network Weight Update Suppression for Transfer Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep learning techniques face high calculation costs and catastrophic forgetting issues when performing transfer learning across tasks, particularly in neural networks used for image recognition and other processing tasks.
Innovation Solution
A method that determines the importance degree of units in a neural network for each task, identifies dissimilar tasks, and suppresses weight parameter updates based on these importance degrees to prevent catastrophic forgetting, while using a low calculation cost approach by clustering feature vectors to determine task similarity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If learning is performed using another learning set to enable recognition of new target objects, then the neural network can recognize more target objects, but the accuracy of distinction among previously learned target objects deteriorates due to catastrophic forgetting
Solution Approach 1:
The patent segments the weight parameters into multiple groups corresponding to different tasks, and applies different update suppression strategies to each group. Specifically, weight parameters are divided into those updated based on current task gradients and those updated based on combined gradients from multiple tasks, allowing selective preservation of knowledge for different task groups while enabling learning of new tasks.
Solution Approach 2:
The patent changes the update rule for weight parameters by introducing task-specific learning rates and gradient combination strategies. The update suppression mechanism modifies the standard gradient descent update by scaling gradients from different tasks differently, thereby controlling the degree of parameter change to prevent catastrophic forgetting while enabling new task acquisition.
2Reliability
If the HAT technique is used to suppress catastrophic forgetting, then task distinction accuracy is maintained, but the calculation cost becomes extremely high
Solution Approach 1:
The patent extracts and applies gradient update suppression selectively to specific groups of weight parameters rather than uniformly to all parameters. By identifying which task groups require protection from forgetting and applying suppression only to those specific parameter groups, the calculation overhead is significantly reduced compared to applying the technique globally to the entire neural network.
Solution Approach 2:
The patent applies gradient update suppression partially to only those weight parameters that are critical for maintaining performance on previous tasks, rather than applying it excessively to all parameters. This selective application reduces the computational burden while still achieving the goal of preventing catastrophic forgetting for important tasks.
Data Source
AI summary
According to an aspect of the present disclosure, a method executed by an information processing apparatus in order to cause a neural network to learn a Tth task corresponding to a Tth learning set is provided. The method includes: for each of a plurality of units, determining an importance degree of the unit in the Tth task; for each of a plurality of layers, determining dissimilar tasks from among a first task to a (T−1)th task, the dissimilar tasks being not similar to the Tth task in terms of behaviors in the layer; and in learning that uses the Tth learning set, suppressing updating of weight parameters of the plurality of units included in the plurality of layers in accordance with importance degrees in the dissimilar tasks determined for each of the plurality of layers.