Multi-GPU Deep Learning Training Using CPU Gradient Synchronization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current deep learning neural network training methods using multiple GPUs face scalability limitations due to the inefficiencies in exchanging and accumulating gradients, particularly in the backward phase, which slows down the training process.
Innovation Solution
The proposed method employs a CPU-GPU data parallel training approach that forms chunks of neural network layers and performs chunk-wise synchronization operations during the backward phase, utilizing CPUs to accelerate gradient accumulation and communication, thereby reducing synchronization bottlenecks and optimizing training performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If data parallelism is used for training a deep neural network on multiple GPUs, then simplicity is achieved, but scalability is bound by data transfers for exchanging and accumulating gradients among the GPUs
Solution Approach 1:
The patent divides the neural network into multiple stages, with each GPU responsible for specific stages. This segmentation allows different GPUs to work on different portions of the network simultaneously, reducing the frequency of gradient synchronization operations and improving scalability while maintaining data parallelism simplicity.
Solution Approach 2:
The patent introduces a host CPU as an intermediary to manage gradient accumulation and synchronization between GPUs. The CPU acts as a mediator that collects gradients from multiple GPUs, accumulates them, and redistributes updated parameters, thereby reducing the direct communication overhead between GPUs and improving system scalability.
2Reliability
If gradient exchange and accumulation operations are performed among multiple GPUs during backward phase, then training accuracy is maintained, but training speed is reduced due to synchronization bottlenecks
Solution Approach 1:
By segmenting the neural network into multiple stages assigned to different GPUs, the patent reduces the frequency of gradient synchronization operations during the backward phase. Each GPU can complete forward and backward propagation for its assigned stages without requiring intermediate synchronization, thereby maintaining training accuracy while significantly improving training speed.
Solution Approach 2:
The patent performs gradient accumulation on the host CPU in advance before distributing updated parameters back to GPUs. This preliminary action allows GPUs to continue processing without waiting for synchronous gradient exchanges during the backward phase, thereby maintaining accuracy through proper gradient accumulation while improving training speed by eliminating synchronization bottlenecks.
Data Source
AI summary
A computer-implemented method, computer program product, and computer processing system are provided for accelerating neural network data parallel training in multiple graphics processing units (GPUs) using at least one central processing unit (CPU). The method includes forming a set of chunks. Each of the chunks includes a respective group of neural network layers other than a last layer. The method further includes performing one or more chunk-wise synchronization operations during a backward phase of the neural network data parallel training, by each of the multiple GPUs and the at least one CPU.