Neural Network Training Scheduling for Bandwidth-Aware Parallelism
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network training processes face inefficiencies in utilizing network bandwidth when combining data parallelism and pipeline parallelism, leading to suboptimal training speed.
Innovation Solution
An information processing system that employs data and pipeline parallelism, utilizing a scheduling method to optimize the execution order of forward and backward calculations, as well as ReduceScatter and Allgather processes across multiple workers, ensuring effective utilization of network bandwidth.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data parallelism and pipeline parallelism are combined to improve training speed, then the training efficiency is improved, but the network bandwidth utilization becomes inefficient
Solution Approach 1:
The scheduling unit performs preliminary scheduling of communication operations before they execute. It determines the execution order of ReduceScatter and Allgather operations across different pipeline stages, ensuring that workers with available network bandwidth are selected for communication first. This preliminary planning prevents network congestion and optimizes bandwidth utilization during the training process.
Solution Approach 2:
The system dynamically adjusts the scheduling of communication operations based on real-time conditions. The scheduling unit monitors network bandwidth availability and worker states, then dynamically determines which workers should perform ReduceScatter or Allgather operations first. This dynamic adaptation allows the system to optimize network bandwidth utilization while maintaining training speed.
2Productivity
If scheduling is optimized to improve network bandwidth utilization, then the training efficiency is improved, but the scheduling complexity increases
Solution Approach 1:
The scheduling process is segmented into distinct phases: determining communication operations, selecting workers based on availability, and ordering operations across pipeline stages. This segmentation allows the complex scheduling problem to be broken down into manageable steps, reducing the computational overhead while maintaining optimization effectiveness.
Solution Approach 2:
The scheduling unit incorporates feedback mechanisms to monitor network bandwidth utilization and worker execution states. Based on this feedback, the system adjusts the scheduling decisions in real-time, ensuring optimal bandwidth utilization. The feedback loop allows the system to learn from previous scheduling decisions and improve future scheduling without requiring overly complex algorithms.
Data Source
AI summary
An information processing system includes one or more first processors and one or more second processors that perform a training process of a neural network. The one or more first processors perform forward processing on first and second data, using first parameters, to generate first and second outputs. The one or more second processors perform forward processing based on the first output, using second parameters, to generate a third output; perform forward processing based on the second output, using the second parameters, to generate a fourth output; generate first gradient information of the second parameters based on the third and outputs; perform a first process on the first gradient information; update the second parameters based on a result of the first process; and transmit the updated second parameters to the one or more first processors. The one or more first processors perform a second process, using the updated second parameters.