Gradient Accumulator with Free Momentum for Neural Network Training
Solution Overview
Problem
Current methods for updating weights in neural networks, such as batched stochastic gradient descent (SGD), require additional memory for larger batch sizes and momentum, leading to inefficiencies in memory usage during training.
Innovation Solution
Implementing gradient accumulation with free momentum using accumulators that generate momentum between passes of neural network training, allowing for continuous asynchronous updates and weight updates during back-propagation, without requiring additional memory elements by using a single memory element to approximate a moving average via a 'leaky' accumulator.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Stability of the object's composition
If batched stochastic gradient descent with larger batch sizes is used, then training stability is improved, but additional memory is required
Solution Approach 1:
The patent combines gradient accumulation and momentum computation into a single memory element (the accumulator). The accumulator stores both the accumulated gradient information and the momentum term, eliminating the need for separate memory allocations. This merging allows larger effective batch sizes for improved training stability without proportionally increasing memory usage.
Solution Approach 2:
The accumulator serves multiple functions simultaneously: it accumulates gradients across batches, computes momentum terms, and stores weight updates. This multi-functionality allows the system to achieve the benefits of large batch training (stability) while using the same memory resources efficiently for multiple purposes.
2Productivity
If gradient accumulation with momentum is implemented, then convergence speed is improved, but extra memory elements are required
Solution Approach 1:
The patent merges the momentum computation and gradient accumulation functions into a single accumulator structure. The accumulator maintains both the running sum of gradients and the momentum term using the same memory storage, thereby achieving faster convergence through momentum without requiring additional separate memory elements.
Solution Approach 2:
The accumulator is designed to automatically compute momentum terms from its own stored gradient accumulation data. The momentum computation is performed in-place within the accumulator structure, eliminating the need for external memory elements to store momentum information separately.
3Productivity
If on-device gradient accumulation with block average is used, then pipeline modeling efficiency is improved, but additional memory for momentum is required
Solution Approach 1:
The patent combines the block average gradient accumulation with momentum computation in a unified accumulator structure. The accumulator processes gradients in blocks (improving pipeline efficiency) while simultaneously maintaining momentum terms within the same memory footprint, avoiding additional memory requirements.
Solution Approach 2:
The accumulator dynamically adjusts its computation to work with block-averaged gradients from pipeline processing. It accumulates gradients in blocks, computes averages, and simultaneously updates momentum terms, allowing the system to maintain high pipeline efficiency while using memory efficiently for both accumulation and momentum.
Data Source
AI summary
Methods for gradient accumulation with free momentum are performed by systems and devices during neural network model training. An accumulator that includes a processor circuit and a memory element generates free momentum between passes of a neural network model training process. The processor circuit receives a difference weight (gradient) and generates a first input by applying a weighting parameter thereto. The processor circuit obtains a prior weight from the memory element and generates a second input by applying another weighting parameter thereto. The processor circuit generates a filtered input with momentum by filtering the first and second input. The memory element generates a stored next pass weight by accumulating the filtered input with the prior weight. A computing resource then processes the next pass of the neural network model training using the stored next pass weight. The methods, systems, and devices are applicable to pipelined model parallelism training processes.