Hardware-Aware LLM Training Scheduling on Heterogeneous GPU Clusters

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Heterogeneous computing systems face inefficiencies due to varying hardware capabilities across nodes, leading to underutilization of high-capability nodes and overburdening of low-capability nodes, and inefficient data sharing using Remote Direct Memory Access (RDMA).

Innovation Solution

Implement hardware-aware scheduling that recognizes and optimizes node capabilities, utilizing cache-coherent interconnect memories like Compute Express Link (CXL) for direct memory access and optimizing data distribution across nodes.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If homogeneous hardware assumption is used in training systems, then system simplicity is maintained, but heterogeneous computing systems suffer from low training efficiency

Engineering Contradiction:

Improvetraining efficiencyVSAvoidhardware heterogeneity management

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The system dynamically adjusts workload parameters (batch size, micro-batch configuration, data partitioning) based on the specific hardware capabilities of each node. The scheduler modifies these parameters to match node performance characteristics, transforming the fixed homogeneous scheduling approach into an adaptive heterogeneous scheduling system that optimizes training efficiency across diverse hardware configurations

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The scheduling system transitions from static homogeneous workload distribution to dynamic heterogeneous scheduling. It continuously monitors node performance metrics and adjusts workload allocation in real-time, enabling the system to adapt to varying hardware capabilities and optimize resource utilization across the heterogeneous cluster throughout the training process

Inventive Principle:

Principle #15Dynamics

2Productivity

If workload is distributed without hardware awareness, then high-capability nodes are underutilized, but low-capability nodes become overburdened

Engineering Contradiction:

Improvenode utilization efficiencyVSAvoidworkload distribution balance

Core Design Contradiction:

ProductivityVSEase of operation

Solution Approach 1:

The system applies differentiated workload characteristics to different nodes based on their local hardware qualities. Each node receives workloads tailored to its specific computational capabilities, memory bandwidth, and storage performance. This local optimization ensures that high-capability nodes handle more intensive tasks while low-capability nodes manage lighter workloads, maximizing overall cluster utilization without manual intervention

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The scheduling system implements feedback mechanisms that monitor node performance metrics during training operations. Based on this feedback, the scheduler dynamically adjusts workload distribution to maintain balanced utilization. Nodes that are underutilized receive additional workload, while overburdened nodes have their workload reduced, creating a self-regulating system that optimizes resource allocation across the heterogeneous cluster

Inventive Principle:

Principle #23Feedback

3Speed

If RDMA is used for data sharing, then data transfer speed is improved, but data transfer bottlenecks still occur in heterogeneous systems

Engineering Contradiction:

Improvedata transfer speedVSAvoiddata transfer efficiency

Core Design Contradiction:

SpeedVSProductivity

Solution Approach 1:

The system performs preliminary data preprocessing and partitioning actions before data transfer begins. Workloads are pre-segmented into micro-batches with optimized data layouts that match the capabilities of destination nodes. This preliminary preparation reduces the complexity of data transfer operations and minimizes bottlenecks during actual transmission, even in heterogeneous environments with varying RDMA capabilities

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The scheduling system segments data workloads into smaller micro-batches that can be efficiently transferred and processed. This segmentation allows for finer-grained control over data distribution, enabling the system to optimize transfer sizes and frequencies based on the specific RDMA capabilities of different node pairs. Smaller, more frequent transfers reduce memory bandwidth bottlenecks and improve overall data transfer efficiency

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS20260037307A1Hardware-aware scheduling and data orchestration for balanced LLM training on heterogeneous GPU clusters

Publication Date: 2026.02.05 SAMSUNG ELECTRONICS CO LTD

AI summary

A scheduling system is disclosed. The scheduling system may include a simulator to process information regarding a heterogeneous computing system. An intra-node scheduler may determine whether individual nodes should use a tensor parallel approach or a data parallel approach. An inter-node scheduler may schedule operations between the nodes. An evaluator may evaluate a performance of the heterogeneous computing system.