Hybrid Work Donation and Stealing for GPU Load Balancing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for load balancing in processing devices, such as GPUs, suffer from inefficiencies due to high software thread contention and overhead costs, particularly in high thread count situations, leading to suboptimal performance.
Innovation Solution
A hybrid work donation and stealing technique is employed, where queue elements are distributed between processing cores using a combination of work donation and work stealing, with atomic operations ensuring synchronized data access within processing cores and across the device, to maintain load balance and distribute workloads efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If work donation process is used to transfer unprocessed workloads between execution units within a processing core, then load imbalances among execution units are alleviated, but high software thread contention occurs in high thread count situations
Solution Approach 1:
The patent segments the workload distribution mechanism into two distinct pathways: work donation for intra-core load balancing and work stealing for inter-core load balancing. This segmentation allows each mechanism to operate independently with optimized data structures (workgroup queues for donation, core-level queues for stealing), reducing contention by preventing both mechanisms from competing for the same resources simultaneously.
Solution Approach 2:
The patent introduces an intermediary mechanism where processing cores submit unprocessed workloads to a shared pool, and execution units can steal from this pool when their local workgroup queues are depleted. This intermediary shared pool acts as a buffer that reduces direct contention between workgroups while maintaining load balancing effectiveness.
2Productivity
If work stealing is used to transfer queue elements between processing cores, then load imbalances among processing cores are improved, but overhead costs increase
Solution Approach 1:
The patent implements preliminary action by having processing cores proactively submit unprocessed workloads to the shared pool before execution units need them. This advance preparation reduces the overhead of work stealing by ensuring that stolen work is already prepared and available, minimizing the time and resources needed for inter-core workload transfer.
Solution Approach 2:
Execution units autonomously monitor their own workload status and independently initiate work stealing operations when their queues are depleted, without requiring centralized scheduling intervention. This self-service approach reduces overhead by eliminating the need for complex inter-core coordination protocols while maintaining effective load balancing.
3Reliability
If atomic operations are used to synchronize data access within processing cores, then data integrity is maintained, but processing speed is reduced
Solution Approach 1:
The patent applies local quality by implementing atomic operations only where data integrity is critical (at the boundaries of workgroup queue operations and shared pool access), while allowing non-atomic, faster operations for internal workgroup processing. This selective application of atomic operations maintains data integrity where needed while preserving processing speed in less critical paths.
Data Source
AI summary
Briefly, methods and apparatus to rebalance workloads among processing cores utilizing a hybrid work donation and work stealing technique are disclosed that improve workload imbalances within processing devices such as, for example, GPUs. In one example, the methods and apparatus allow for workload distribution between a first processing core and a second processing core by providing queue elements from one or more workgroup queues associated with workgroups executing on the first processing core to a first donation queue that may also be associated with the workgroups executing on the first processing core. The method and apparatus also determine if a queue level of the first donation queue is beyond a threshold, and if so, steal one or more queue elements from a second donation queue associated with workgroups executing on the second processing core.