Heterogeneous LLM Training Load Balancing With Shared Memory Pools

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Heterogeneous computing systems face inefficiencies due to varying hardware capabilities and memory capacities across nodes, leading to unequal workload completion times and inefficient data sharing using Remote Direct Memory Access (RDMA).

Innovation Solution

Implement hardware-aware scheduling and leverage cache-coherent interconnect memories like Compute Express Link (CXL) to optimize node operations and facilitate direct access to a common memory pool, reducing reliance on RDMA for data transfer.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If hardware-aware scheduling is implemented in heterogeneous computing systems, then system efficiency and productivity are improved, but device complexity and scheduling algorithm complexity increase

Engineering Contradiction:

Improvetraining efficiencyVSAvoidscheduling system complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The system performs preliminary characterization of hardware capabilities before scheduling training operations. The scheduling mechanism pre-assesses node characteristics including processor types, memory capacities, and interconnect bandwidths, then uses this pre-acquired information to make informed scheduling decisions, avoiding real-time complexity while maintaining optimization

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

A scheduling mechanism acts as an intermediary layer between the heterogeneous hardware nodes and the training workloads. This intermediary abstracts the complexity of hardware variations by introducing intermediate variables that represent effective computing speeds and memory capacities, simplifying the scheduling decision-making process while still accounting for hardware differences

Inventive Principle:

Principle #24Intermediary (Mediator)

2Speed

If RDMA is used for data transfer between nodes, then data sharing speed is improved, but system efficiency deteriorates due to unequal workload completion times

Engineering Contradiction:

Improvedata transfer speedVSAvoidtraining efficiency

Core Design Contradiction:

SpeedVSProductivity

Solution Approach 1:

The system dynamically adjusts workload distribution based on real-time or pre-assessed node performance characteristics. Rather than static load balancing, the scheduling mechanism modifies task allocation to account for varying computing speeds and memory capacities across nodes, ensuring that faster nodes handle proportionally more work to maintain synchronized completion

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The scheduling mechanism applies localized optimization to each node based on its specific hardware characteristics. Each node's workload is tailored to its local capabilities—processors, memory, and interconnect speeds—rather than applying a uniform distribution strategy, thereby optimizing overall system productivity while maintaining efficient data transfer

Inventive Principle:

Principle #3Local quality

Data Source

PatentUS20260037340A1Memory aware load balancing mechanism for heterogeneous LLM training

Publication Date: 2026.02.05 SAMSUNG ELECTRONICS CO LTD

AI summary

A heterogeneous computing system is described. The heterogeneous computing system may include nodes, which may include processing elements, the processing elements including local memories. The nodes may have capabilities that may differ. The heterogeneous computing system may also include a memory pool, accessible to the processing elements.