Logit Adjustment for Machine Learning Gradient Saturation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models face challenges in accurately classifying objects in images due to gradient saturation issues during training, leading to decreased recognition accuracy and slower learning speeds.
Innovation Solution
A learning device and method that adjusts the output values of machine learning models by adding penalties to logits for non-correct-answer classes and subtracting from correct-answer classes, using a predetermined parameter to increase the absolute value of gradients and prevent saturation, thereby improving training efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If gradient scaling or selection methods are used to improve long-tailed object detection, then recognition accuracy of small-sample classes is improved, but monitoring of gradient information is required making processes troublesome
Solution Approach 1:
The patent changes the parameter space by operating on logit values directly rather than gradient values. By adding a logit adjustment value to the logit output before softmax calculation, the method achieves gradient effect without needing to monitor or manipulate gradient information, thus improving accuracy while simplifying the process
Solution Approach 2:
The patent introduces an intermediary component (logit adjustment value) between the logit output and the final classification. This intermediary allows control over the classification outcome without directly manipulating gradients, avoiding the need for gradient monitoring while still achieving the desired effect on recognition accuracy
2Measurement precision
If gradient of specific logits is ignored to improve small-sample class recognition, then recognition accuracy of small-sample classes is improved, but recognition accuracy of classes with ignored gradients may deteriorate
Solution Approach 1:
The patent applies local quality by adjusting logit values specifically for certain classes (small-sample classes) while leaving other classes unchanged. The logit adjustment value is configured to affect only specific target classes, thereby improving their recognition accuracy without negatively impacting the recognition accuracy of other classes
3Device complexity
If conventional training methods are used, then training process is simple, but gradient saturation occurs leading to decreased learning speed and accuracy
Solution Approach 1:
The patent performs preliminary action by adjusting the logit values before the softmax function is applied. This pre-adjustment ensures that the input to softmax has appropriate magnitude characteristics, preventing gradient saturation from the outset and maintaining high learning speed throughout training without complex additional processing
Data Source
AI summary
A learning device includes a storage medium configured to store computer-readable instructions and a processor connected to the storage medium. The processor trains a machine learning model that receives an input of an image including a plurality of pixels and outputs a degree of accuracy with which each pixel corresponds to a class indicating a type of an object by executing the computer-readable instructions. The processor adjusts an output value of the degree of accuracy using a predetermined parameter with a tendency to decrease the output value of the degree of accuracy corresponding to a correct-answer class and to increase the output value of the degree of accuracy corresponding to a class other than the correct-answer class and trains the machine learning model on the basis of the adjusted output value.