Transformer Parallel Scheduling for Dynamic AI Workload Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The escalating size and complexity of generative AI models necessitate effective parallelism for distributed execution, but existing methods face challenges such as scalability issues, suboptimal performance, and the dynamic nature of input/output shapes, particularly in transformer models.
Innovation Solution
An automatic partition framework that leverages repeated layer structures in transformer models, generating candidate execution plans using various parallelisms, and simulating resource usage to identify an optimal partitioning scheme, accommodating dynamic batching and varying input/output sizes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If distributed execution is used to manage computational power requirements, then the model can be executed on available devices, but developing effective parallelism for partitioning devices becomes complex
Solution Approach 1:
The system segments the complex task of parallelism development into automated components. It divides the model into discrete operations and automatically generates parallel execution plans, eliminating the need for manual parallelism development while enabling distributed execution across multiple devices.
Solution Approach 2:
The system implements self-service by automatically generating candidate execution plans and selecting optimal parallel schedules without requiring expert intervention. The automated plan generation and selection process enables the system to manage its own parallelism requirements, reducing development complexity.
2Productivity
If manual parallelism development is used, then execution plans can be created, but the process is time-consuming and may not explore the full search space
Solution Approach 1:
The system creates multiple candidate execution plans by copying and varying parallel schedule configurations. It generates numerous candidate plans with different partitioning strategies and automatically evaluates them, enabling comprehensive search space exploration without manual effort and significantly improving plan generation speed.
3Loss of time
If the search space is constrained to reduce evaluation time, then fewer candidate plans are evaluated, but the quality of the optimal execution plan may deteriorate
Solution Approach 1:
The system dynamically adjusts the evaluation process by using automated simulation to quickly assess candidate plans. It prioritizes evaluation of promising candidates while maintaining the ability to explore the full search space, balancing evaluation time with execution plan quality through adaptive resource allocation.
Solution Approach 2:
The system changes evaluation parameters by simulating different execution scenarios and resource allocations. It adjusts simulation parameters to efficiently evaluate candidate plans, maintaining high evaluation speed while ensuring thorough assessment of plan quality through multiple simulation iterations.
4Ease of manufacture
If static partitioning is used, then device assignment is simple, but it cannot adapt to dynamic computational resources and varying workload requirements
Solution Approach 1:
The system implements dynamic partitioning by automatically generating execution plans that adapt to current resource availability and workload characteristics. It re-evaluates and regenerates parallel schedules based on changing conditions, enabling the system to maintain simple partitioning logic while achieving high adaptability to dynamic environments.
Data Source
AI summary
A computer-implemented method can receive an internal representation of a transformer model which defines one or more repeating blocks, each block including a sequence of cells, and each cell including a set of tasks of the transformer model. The method can search for a plurality of parallel schedules for partitioning devices included in a device cluster for parallel execution of the transformer model. The searching includes determining a number of model replicas, determining a number of stages that divide the one or more repeating blocks, determining a number of cell replicas for each cell in a block, and for each cell replica of a cell, generating a task mapping which maps the set of tasks included in the cell to devices partitioned into the cell replica.