Angle-Based Stochastic Gradient Descent for Convergence
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing stochastic gradient descent (SGD) optimization algorithms face challenges in accurately computing new gradients due to deviations between mini-batches, leading to accumulation of pattern deviations and convergence issues in local minima.
Innovation Solution
The angle-based stochastic gradient descent (AG-SGD) algorithm determines new gradients by calculating the inner angle between current and previous gradients, adjusting weights, and modifying the learning rate based on this angle to reduce pattern deviations and enhance convergence.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing SGD optimization algorithms use original past gradient (OPG) to improve new gradient accuracy, then gradient combination is simplified, but pattern deviation accumulates and causes convergence to local minima
Solution Approach 1:
The patent transforms the past gradient vector by rotating it according to the angle θ calculated from the dot product of current and past gradients. This parameter transformation ensures that the rotated past gradient aligns better with the current gradient direction, reducing pattern deviation while maintaining the benefits of gradient combination. The rotation angle is dynamically adjusted based on the relationship between current and past gradients, allowing the system to adapt to changing optimization landscapes.
2Measurement precision
If SGD combines current gradient and past gradient to eliminate mini-batch deviation, then gradient accuracy improves, but computational complexity increases
Solution Approach 1:
The patent performs preliminary calculations of the angle θ and rotation transformation before combining gradients. By pre-computing the rotation matrix based on the angle between current and past gradients, the system prepares the transformed past gradient in advance, making the final gradient combination step more efficient. This preliminary action reduces the computational burden during the main optimization loop.
3Measurement precision
If SGD uses more computing resources for accurate gradient computation, then training accuracy improves, but training time increases
Solution Approach 1:
The patent dynamically adjusts the gradient combination strategy based on the angle θ between current and past gradients. When θ indicates strong alignment, the system can rely more on the simpler combination method. When θ indicates significant deviation, the system applies the rotation transformation to correct the pattern deviation. This dynamic adaptation allows the system to maintain accuracy while reducing unnecessary computational overhead in favorable conditions.
Data Source
AI summary
Systems and methods for optimizing and/or solving objective functions are provided. Angle-based stochastic gradient descent (AG-SGD) can be used to alleviate pattern deviation(s) not resolved by related art systems and methods. AG-SGD can use the angle between the current gradient (CG) and the previous gradient (PG) to determine the new gradient (NG).