Distributed Model Training With Overlapped Communication and Computation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed model training schemes face high communication overhead and low efficiency, especially in large-scale deep neural network training, making them costly and difficult to implement in clusters with limited bandwidth, and they have poor expandability.
Innovation Solution
A distributed model training method that divides the training process into internal and external iterations, allowing asynchronous ALLReduce communication operations to overlap with computation, using stale model parameter values for external iterations, and optimizing momentum updates to improve efficiency and adaptability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If small-batch parallel optimization method is adopted for distributed model training, then training efficiency is improved, but communication overhead increases and cluster construction cost increases
Solution Approach 1:
The patent segments the training process into multiple stages: local training stage where each node performs independent training, and global synchronization stage where parameter aggregation occurs. This segmentation allows computation and communication to be separated in time, reducing the frequency and size of communication operations while maintaining training efficiency.
Solution Approach 2:
The patent performs preliminary local training for multiple iterations before each global synchronization. By completing local training updates in advance, the system maximizes computation before communication, reducing the relative impact of communication overhead on total training time and improving overall productivity.
2Loss of energy
If high-speed communication interconnectivity is constructed, then communication overhead is reduced, but cluster construction cost increases
Solution Approach 1:
The patent changes the parameter of communication frequency by performing global synchronization less frequently, aggregating parameters only after multiple local training iterations. This parameter change allows the system to achieve good communication efficiency without requiring expensive high-speed interconnects, as communication occurs less often with smaller data volumes.
3Productivity
If existing distributed model training scheme is applied to large distributed computing node cluster, then training scalability is improved, but bandwidth requirements increase and expandability deteriorates
Solution Approach 1:
The patent implements a dynamic training schedule where the number of local training iterations between global synchronizations can be adjusted based on cluster size and bandwidth conditions. This dynamic adjustment allows the same basic framework to adapt to clusters ranging from small to large scales without requiring proportional increases in bandwidth infrastructure, thereby improving expandability.
Data Source
AI summary
A distributed model training method includes that: the training of an internal iteration with a preset number of internal iterations is performed on a preset model through a computation process, to obtain a first node model parameter value of a current global iteration; a second node model parameter value of the current global iteration of a second computing node in a distributed system is acquired through a communication process running in parallel with the computation process, and a first ALLReduce model parameter value of the current global iteration is determined according to the first node model parameter value and the second node model parameter value; and external iteration is performed through the computation process by using a second ALLReduce model parameter value of a last global iteration, to obtain a target model parameter value of the current global iteration.