Pipeline Parallelism Compilation for Automated Micro-Batch Scheduling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for training deep learning models require additional user efforts in programming and system settings for pipeline parallel computing, such as explicitly determining micro-batches and communication processes, which are inconvenient and inefficient.
Innovation Solution
A method and apparatus for pipeline parallelism compiling that automatically determines operation groups and sets, allocates accelerators, and sequences processing to optimize parallelism, reducing the need for manual user intervention and enhancing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If users manually determine each stage or micro-batch of the pipeline and explicitly insert communication processes, then control over the parallel processing is precise, but the programming effort and complexity increase significantly
Solution Approach 1:
The compiler automatically analyzes the source program, determines operation groups, generates micro-batches, and allocates accelerators without requiring user intervention. The system self-manages the pipeline parallelism configuration, eliminating the need for users to manually determine stages or insert communication processes while maintaining precise control through automated optimization
Solution Approach 2:
The compiler performs preliminary analysis of the source program to determine operation groups and their dependencies before generating the parallel execution plan. This preliminary action includes analyzing the computational graph, identifying independent operations, and pre-configuring the pipeline stages, so that when execution occurs, the pipeline is already optimized and ready without requiring user setup
2Productivity
If additional efforts are required during programming for pipeline parallelism, then the parallel processing can be customized, but the development time and user burden increase
Solution Approach 1:
The manual mechanical process of dividing pipelines into stages and configuring communication processes is replaced by an automated compiler system. The compiler mechanically analyzes the source program, generates operation groups, creates micro-batches, and allocates resources automatically, substituting user manual configuration with automated compilation that achieves the same productivity benefits without the time cost
3Productivity
If the pipeline is divided into more micro-batches for parallel processing, then the throughput increases, but the memory usage and live range of activation tensors increase
Solution Approach 1:
The pipeline is segmented into operation groups based on computational dependencies, and each group is further divided into micro-batches that can be processed in parallel. The compiler segments the computational graph into independent operation groups, then allocates micro-batches to accelerators in a way that minimizes memory overlap and activation tensor live ranges while maintaining high throughput
Solution Approach 2:
The compiler dynamically determines the optimal number of micro-batches and their allocation to accelerators based on the specific computational characteristics of the source program. This dynamic configuration allows the system to adjust the balance between throughput and memory usage for different workloads, optimizing performance rather than using a fixed approach
Data Source
AI summary
A method for pipeline parallelism compiling is provided, which is executed by one or more processors, and includes receiving a source program associated with training of a machine learning model, determining, based on the source program, a plurality of operation groups including operations executed on training data of the machine learning model, generating a plurality of micro-batches from the training data, and determining, for each of the plurality of micro-batches, a plurality of operation sets corresponding to the plurality of operation groups.