Hybrid Parallel Model Training to Reduce Pipeline Idle Time
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The inefficiency of computing node utilization in training deep neural networks due to idle time caused by the sequential nature of pipeline parallelism in deep neural network training, leading to increased training duration.
Innovation Solution
Implement a hybrid parallelism manner that combines data and pipeline parallelism by dividing the neural network into pipeline stages with consecutive sub-stages, allowing computing nodes to proceed with forward and backward propagation without waiting for previous stages to complete, and utilizing fine-grained division and communication synchronization to reduce idle time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If pipeline parallelism is used to divide the neural network into multiple pipeline stages, then the model can be trained using distributed computing, but computing nodes have to wait for previous stages to complete, causing idle time and low utilization
Solution Approach 1:
The patent divides each pipeline stage into multiple sub-stages (fine-grained segmentation), allowing computing nodes to process different sub-stages in an interleaved manner. This segmentation enables better overlap of computation and communication, reducing idle time while maintaining the benefits of pipeline parallelism for distributed training.
2Reliability
If pipeline stages are executed sequentially, then forward propagation can be performed correctly, but the computing node for the next pipeline stage must wait, reducing overall training speed
Solution Approach 1:
The patent performs preliminary actions by pre-computing and caching intermediate results from previous sub-stages before they are needed by subsequent pipeline stages. This allows computing nodes to prepare data in advance, eliminating waiting time while ensuring that forward propagation receives the correct intermediate results when needed.
Solution Approach 2:
The patent introduces dynamic scheduling of sub-stages across pipeline stages, where the execution order and timing of sub-stages are adjusted based on readiness of intermediate results. This dynamic approach allows the system to maintain correctness of forward propagation while optimizing the speed of training by reducing idle waiting time.
3Productivity
If fine-grained division into sub-stages is implemented, then utilization of computing nodes is improved, but the complexity of coordinating multiple sub-stages across pipeline stages increases
Solution Approach 1:
The patent implements feedback mechanisms where computing nodes report the completion status of sub-stages and the readiness of intermediate results to a coordination system. This feedback enables automatic scheduling and coordination of sub-stages across pipeline stages, reducing the manual complexity of coordination while maintaining high utilization of computing nodes.
Data Source
AI summary
This application discloses a model training method and apparatus based on a hybrid parallelism manner. In this method, a neural network model is divided into a plurality of pipeline stages, and each pipeline stage includes a plurality of sub-stages of the neural network model. Computing nodes corresponding to the plurality of pipeline stages are invoked in a hybrid parallelism manner according to a sequence of sub-stages in the neural network model. When iterative training is performed on a network layer in a corresponding pipeline stage, because sub-stages at same locations in adjacent pipeline stages are consecutive in the neural network model, the computing node does not need to wait for completion of forward propagation of a previous pipeline stage, and can perform forward propagation on the corresponding pipeline stage only after forward propagation of the 1st sub-stage in the previous pipeline stage is completed.