Parallel Neural Network Training Overlap Gradient Accumulation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for training neural networks on multiple graphics processing units face significant communication overhead, particularly when dealing with large models and numerous parameters, which limits the scalability of data parallelism.
Innovation Solution
A training module that overlaps backpropagation and gradient transfer processes by collecting and accumulating gradients on CPUs during the backward phase, allowing for efficient parameter updating on GPUs, thereby reducing communication overhead and enhancing training efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If gradient accumulation and parameter updating are performed sequentially on multiple GPUs, then training accuracy is maintained, but communication overhead and training time increase significantly
Solution Approach 1:
The patent merges gradient accumulation and parameter updating operations into a single synchronized step performed on a single GPU (the master GPU). Instead of each GPU independently accumulating gradients and updating parameters, all worker GPUs send their gradients to the master GPU, which accumulates them and performs a single parameter update. This consolidation reduces the number of communication rounds and synchronization points, thereby reducing communication overhead while maintaining training accuracy through centralized gradient aggregation.
Solution Approach 2:
The master GPU acts as an intermediary between worker GPUs and the parameter storage. Worker GPUs send their computed gradients to the master GPU, which then accumulates these gradients and updates the model parameters. This intermediary approach centralizes the communication and coordination overhead on a single node, allowing worker GPUs to independently compute gradients without frequent synchronization, thus reducing overall communication overhead while ensuring all GPUs use consistent updated parameters.
2Productivity
If data parallelism is scaled up across numerous GPUs for large models, then training capacity increases, but communication overhead becomes the limiting factor
Solution Approach 1:
The patent segments the training workload by designating one GPU as a master node and the others as worker nodes. Worker GPUs independently compute gradients for their assigned data batches, while the master GPU handles gradient accumulation and parameter updating. This segmentation allows the system to scale to numerous GPUs without proportionally increasing communication overhead, as only the master GPU needs to receive and aggregate gradients from all workers in each synchronization step, rather than requiring full mesh communication between all GPU pairs.
Data Source
AI summary
A system and method provides efficient parallel training of a neural network model on multiple graphics processing units. A training module reduces the time and communication overhead of gradient accumulation and parameter updating of the network model in a neural network by overlapping processes in an advantageous way. In a described embodiment, a training module overlaps backpropagation, gradient transfer and accumulation in a Synchronous Stochastic Gradient Decent algorithm on a convolution neural network. The training module collects gradients of multiple layers during backpropagation of training from a plurality of graphics processing units (GPUs), accumulates the gradients on at least one processor and then delivers the gradients of the layers to the plurality of GPUs during the backpropagation of the training. The whole model parameters can then be updated on the GPUs after receipt of the gradient of the last layer.