Learning Apparatus Gradient Variance Update Neural Network
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning methods using parallel processing face a decrease in learning effect when the data size (batch size) increases, leading to inefficiencies in neural network parameter updates.
Innovation Solution
A learning apparatus that generates partial data from mini-batches, calculates partial gradients, and updates parameters based on the average value and variance of these gradients, using an overall gradient formula that incorporates the reciprocal of the variance, to adjust learning speeds and improve efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If parallel learning is used to speed up learning processing, then processing speed is improved, but learning effect decreases when batch size increases
Solution Approach 1:
The patent divides the mini-batch into multiple pieces of partial data, where each piece is processed by a different processor. This segmentation allows parallel processing to speed up computation while maintaining effective learning by processing smaller, more manageable data portions that preserve gradient accuracy.
Solution Approach 2:
The patent introduces a variance parameter to modify the gradient update process. By incorporating variance information into the parameter update formula, the system adjusts learning speeds dynamically to compensate for the reduced learning effect caused by parallel processing with smaller batch sizes.
2Reliability
If batch size is increased to improve learning effect, then learning accuracy is improved, but processing time increases
Solution Approach 1:
The patent segments the batch into partial data pieces that can be processed simultaneously by multiple processors. This allows the system to maintain effective learning accuracy through coordinated parallel processing while reducing total processing time compared to sequential processing of the entire batch.
Solution Approach 2:
The patent dynamically adjusts learning speeds based on variance calculations. By making the learning rate adaptive rather than fixed, the system optimizes processing time while maintaining learning accuracy, allowing faster processing when variance is low and more careful updates when variance is high.
Data Source
AI summary
According to one embodiment, a learning apparatus includes processing circuitry. The processing circuitry generates a plurality of pieces of partial data from a mini-batch of learning data used for a plurality of learning processes for learning of a parameter of a neural network using an objective function, calculates a partial gradient that is a gradient related to the parameter of the objective function for each of the pieces of partial data, and updates the parameter based on an average value of the plurality of partial gradients corresponding to the pieces of partial data and a variance for the partial gradients.