Transformer Mixed Parallelism for Scalable Cluster Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The escalating size and complexity of generative AI models necessitate effective parallelism for distributed execution, but existing methods face challenges such as scalability issues, suboptimal performance, and the dynamic nature of input/output shapes, particularly in transformer models.
Innovation Solution
An automatic partition framework that leverages repeated layer structures in transformer models, generating candidate execution plans using mixed parallelisms like pipeline, data, and task parallelisms, with a simulator to estimate resource usage and determine an optimal execution plan.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If distributed execution is used to manage computational power requirements, then the model can be executed on available devices, but developing effective parallelism for partitioning devices becomes complex
Solution Approach 1:
The system segments the transformer model into discrete operations and partitions devices into logical groups, creating a structured framework for distributed execution. This segmentation approach breaks down the complex parallelism problem into manageable components that can be systematically organized and executed
Solution Approach 2:
The system dynamically adapts parallel execution plans based on available device configurations and workload characteristics. By making the parallelism strategy dynamic rather than static, the system can adjust to different device clusters without requiring complex manual reconfiguration, thus improving adaptability while managing complexity
2Power
If more devices are partitioned for parallel execution, then computational power increases, but managing synchronization and interdependencies becomes more difficult
Solution Approach 1:
The system introduces an intermediary execution plan that coordinates communication and synchronization between partitioned devices. This intermediary layer manages the complex interdependencies and synchronization requirements, allowing computational power to scale across multiple devices without proportionally increasing management complexity
3Productivity
If the search space for execution plans is not constrained, then optimal performance can be achieved, but the search space becomes too large to evaluate efficiently
Solution Approach 1:
The system applies local quality constraints by identifying and evaluating only those execution plan characteristics that are locally relevant to specific device configurations and workload types. This approach maintains optimality by focusing evaluation on pertinent factors rather than exhaustively searching the entire search space, thus reducing evaluation time while preserving performance optimization
Data Source
AI summary
A computer-implemented method can generate a parallel schedule for partitioning devices included in a device cluster for parallel execution of a transformer model. The transformer model is represented by a chain of cells. Each cell includes a set of tasks of the transformer model. Generating the parallel schedule includes dividing the chain of cells into one or more sequential stages, creating one or more replicas of the transformer model or some of the cells, and mapping the set of tasks included in a cell to one or more devices of the device cluster. For a given workload, the method can execute the transformer model on the device cluster according to the parallel schedule.