ML Training Pipeline Scheduling With Overlapping Backward Substages
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional machine learning model (MLM) training methods result in a high number of idle processing cycles due to inefficient scheduling of forward and backward passes, particularly in pipeline parallelization, leading to reduced training efficiency.
Innovation Solution
A scheduler identifies substages of passes in MLM training, specifically the activation and weight gradient calculations in backward passes, and schedules them based on data availability rather than completion of the entire pass, allowing for parallel execution and reducing idle cycles.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional multi-instance training is used, then model training can be performed in parallel, but a large number of idle processing cycles are generated
Solution Approach 1:
The backward pass is segmented into two independent sub-stages: activation gradient calculation and weight gradient calculation. This segmentation allows each sub-stage to be executed independently and in parallel with other operations, eliminating the need to wait for the entire backward pass to complete before starting the next microbatch. The segmentation resolves the contradiction by enabling finer-grained parallelism that reduces idle processing cycles while maintaining training throughput.
Solution Approach 2:
The scheduler initiates the backward pass of microbatch N+1 before the forward pass of microbatch N is completely finished. By performing preliminary actions (starting the next backward pass early) based on data availability rather than waiting for complete pass completion, the system reduces idle processing cycles while maintaining efficient multi-instance training throughput.
2Reliability
If forward and backward passes are scheduled sequentially, then data dependency is maintained, but processing efficiency is reduced
Solution Approach 1:
The scheduling system dynamically adjusts the execution order of forward and backward passes based on real-time data availability. Instead of rigid sequential scheduling, the system allows backward passes to overlap with forward passes when data dependencies permit, maintaining correctness while improving efficiency. This dynamic scheduling resolves the contradiction by adapting the execution schedule to actual data readiness rather than following a fixed sequential pattern.
Solution Approach 2:
The scheduler ensures continuous utilization of processing units by initiating backward passes before forward passes complete, eliminating idle cycles. The activation gradient calculation sub-stage can proceed as soon as its input data is available, maintaining continuous useful action without compromising data dependency correctness. This continuity principle resolves the contradiction by keeping processors busy with valid computations rather than waiting for sequential pass completion.
3Reliability
If entire passes are completed before scheduling next operations, then data correctness is ensured, but processing units remain idle
Solution Approach 1:
By segmenting the backward pass into independent sub-stages (activation gradient calculation and weight gradient calculation), the system can schedule and execute sub-stages as soon as their specific data dependencies are met, rather than waiting for the entire backward pass to complete. This segmentation maintains data correctness for each sub-stage while eliminating idle cycles caused by waiting for complete pass completion.
Solution Approach 2:
The scheduler performs preliminary scheduling of backward pass sub-stages before the entire forward pass completes, initiating computations as soon as data is available. This preliminary action ensures data correctness is maintained through proper dependency tracking while preventing processing units from remaining idle during the transition between microbatches.
Data Source
AI summary
A processing system schedules training of a machine learning model based on identifying one or more substages of passes (e.g., backward passes) of microbatches associated with training the machine learning model. At least some of the identified substages for a given layer generate, during a pass, data used to train other layers of the machine learning model, while other substages only generate data used to train the given layer.Accordingly, a scheduler of the processing system schedules the substages based on whether the data generated by the substage is used to train a different layer of the MLM.