Transformer Parallel Execution Planning for Dynamic AI Workloads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The escalating size and complexity of generative AI models necessitate effective parallelism for distributed execution, but existing methods face challenges such as scalability issues, suboptimal performance, and the dynamic nature of input/output shapes, which complicates the distribution of computational workloads across devices.
Innovation Solution
An automatic partition framework that leverages the repeated layer structure of transformer models, generating candidate execution plans using various parallelisms and collective communications, and simulating resource usage to determine an optimal execution plan that accommodates dynamic input/output shapes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If distributed execution is used to manage computational power requirements, then the model can be executed on available devices, but developing effective parallelism for partitioning devices becomes complex
Solution Approach 1:
The system performs self-service by automatically generating candidate execution plans and selecting optimal parallelism strategies without requiring manual intervention. The framework autonomously analyzes the computational graph, identifies partitioning opportunities, and determines the best distribution of operations across devices, thereby resolving the complexity of developing effective parallelism.
Solution Approach 2:
The framework dynamically changes execution parameters by generating multiple candidate execution plans with different parallelism strategies. It evaluates these plans based on various parameters such as device availability, computational load, and data dependencies, then selects the optimal plan that adapts to the current computational resource state.
2Productivity
If manual parallelism development is used, then execution plans can be created, but the process requires significant effort and does not scale well
Solution Approach 1:
The framework performs preliminary action by pre-generating multiple candidate execution plans with different parallelism strategies before actual model execution. It analyzes the computational graph structure, identifies potential partitioning points, and prepares optimized execution plans in advance, thereby eliminating the need for time-consuming manual parallelism development during runtime.
Solution Approach 2:
The system creates multiple copies of candidate execution plans with different parallelism configurations. By generating and evaluating multiple replicated plans with varying device partitions and operation distributions, it efficiently identifies the optimal execution strategy without requiring manual intervention for each configuration.
3Ease of manufacture
If fixed parallelism strategies are used, then implementation is simpler, but the system cannot adapt to dynamic computational resources and varying workload characteristics
Solution Approach 1:
The framework implements dynamic adaptability by generating multiple candidate execution plans with different parallelism strategies and selecting the optimal one based on current computational resource availability and workload characteristics. It dynamically adjusts device partitioning, operation distribution, and data flow configurations to match the prevailing system state, thereby achieving both ease of implementation through automation and adaptability to dynamic conditions.
Data Source
AI summary
A computer-implemented method can receive an internal representation of a transformer model, an internal representation of a device cluster, and an internal representation of a workload for execution of the transformer model on the device cluster. The method can generate a plurality of candidate execution plans based on the internal representation of the transformer model and the internal representation of the device cluster. Each candidate execution plan represents a unique parallel schedule for partitioning devices in the device cluster for parallel execution of the transformer model. The method can determine an optimal execution plan, including evaluating resource usage of the plurality of candidate execution plans based on the internal representation of the workload, and selecting, among the plurality of candidate execution plans, the optimal execution plan which yields the lowest resource usage. The evaluating includes simulating execution of the transformer model on the device cluster to process the workload.