3D Parallelization for Heterogeneous GPU Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing automatic parallelism techniques are unable to efficiently handle heterogeneous GPU clusters, where GPUs with different performance levels are used, due to their complexity and assumption of homogeneous hardware platforms.
Innovation Solution
A method for automatically determining a 3D parallelization policy for training and inference in a heterogeneous GPU cluster system, which involves generating initialization information based on GPU memory capacity, pipeline-parallelizing the model across nodes, and data/tensor-parallelizing layers based on GPU specifications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If existing automatic parallelism techniques are used, then automation of parallelization is achieved, but they fail to handle heterogeneous GPU clusters due to their assumption of homogeneous hardware platforms
Solution Approach 1:
The patent changes the fundamental parameter assumption from homogeneous to heterogeneous hardware platforms. The system dynamically detects and adapts to varying GPU specifications (memory capacity, compute capability) across different nodes, allowing the same automatic parallelization framework to work across diverse hardware configurations without manual intervention or policy determination by the user.
2Reliability
If manual determination of parallelism policy is required, then optimization for specific hardware can be achieved, but usability is greatly harmed
Solution Approach 1:
The system performs self-service by automatically detecting hardware specifications, calculating optimal parallelization policies, and configuring model distribution without requiring user intervention. The parallelization system independently analyzes GPU memory capacities and compute capabilities across nodes, then autonomously determines the best data parallelism, tensor parallelism, and pipeline parallelism configuration, eliminating the need for users to manually specify parallelism policies while still achieving hardware-optimized performance.
3Quantity of substance
If heterogeneous GPU clusters are built to reduce cost, then cost efficiency is improved, but existing automatic parallelism techniques become inapplicable
Solution Approach 1:
The patent creates a universal automatic parallelization framework that can handle both homogeneous and heterogeneous GPU clusters with a single unified approach. The system universally applies the same automatic policy determination process regardless of hardware uniformity, making cost-efficient heterogeneous clusters just another configuration the system can automatically optimize, rather than requiring separate manual configuration approaches.
4Quantity of substance
If GPU servers are built gradually over time, then cost management is improved, but hardware specifications will differ between existing and new devices
Solution Approach 1:
The system introduces dynamic adaptability to handle evolving hardware specifications. As new GPU servers with different specifications are added to the cluster over time, the automatic parallelization system dynamically detects these differences and recalculates optimal policies to accommodate the mixed hardware architecture, allowing cost-managed gradual deployment without sacrificing parallelization effectiveness.
Data Source
AI summary
Disclosed herein is an apparatus and method for 3D parallelization for a heterogeneous GPU cluster. The method may include generating initialization information based on GPU memory capacity in order to parallelize a model across multiple nodes constituting a heterogeneous GPU cluster, pipeline-parallelizing the model based on the multiple nodes using the generated initialization information, and data/tensor-parallelizing layers of the model, which are allocated to each of the multiple nodes according to pipeline parallelization, based on GPUs mounted in the corresponding node.