Persistent GPU Work Assignment for Dynamic Load Balancing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing GPU architectures face challenges in maintaining high core utilization and throughput due to increased communication overhead, idle cores during work assignment, and inefficiencies in managing multi-phase compute tasks, particularly as compute cores become faster and more numerous, leading to suboptimal performance.
Innovation Solution
A new approach decouples the worker (thread block or kernel) from the work item, enabling dynamic load balancing through a programmatic handshake mechanism between workers and a work distributor, allowing workers to request and receive multiple work assignments without reinitialization overhead, and incorporating enhanced hardware capabilities for asynchronous work assignment and load balancing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If work is assigned piece-by-piece to each compute core in traditional GPU architectures, then core occupancy can be maximized, but communication overhead increases and cores experience idle time during work assignment and reinitialization
Solution Approach 1:
The patent applies preliminary action by pre-initializing compute cores before work assignment. Compute cores are kept in a persistent, pre-initialized state ready to execute work items, eliminating the need for reinitialization between work assignments. This allows cores to immediately begin processing new work items upon receipt, significantly reducing idle time and communication overhead while maintaining high core utilization.
2Productivity
If compute cores are increased in number and speed, then processing capacity increases, but communication distance increases leading to longer delays
Solution Approach 1:
The patent introduces an intermediary mechanism where compute cores communicate through a standardized work item interface and shared memory structures rather than direct point-to-point communication. This intermediary layer abstracts and optimizes communication paths, reducing the impact of increased physical distance between cores while maintaining high processing capacity through efficient work distribution and collection mechanisms.
3Productivity
If work items are made smaller to increase parallelism, then more work items can be distributed, but each work item provides insufficient parallelism to saturate compute core throughput
Solution Approach 1:
The patent merges multiple work items into batches that are assigned to compute cores in persistent execution mode. Instead of processing single small work items sequentially, cores process batches of work items continuously without reinitialization, combining the benefits of fine-grained work distribution with sustained compute throughput. This merging approach ensures compute cores remain saturated while maintaining high parallelism across the GPU architecture.
4Ease of operation
If traditional work assignment mechanisms are used, then work can be distributed to cores, but initialization and teardown overhead becomes significant limiting overall throughput
Solution Approach 1:
The patent implements continuity of useful action through persistent compute core execution. Compute cores maintain their initialized state continuously across multiple work item assignments, eliminating the stop-start cycle of traditional architectures. This continuous execution model ensures that initialization overhead occurs only once per compute core lifecycle, while teardown is minimized, significantly reducing overall overhead and improving throughput for work distribution operations.
Data Source
AI summary
In a GPU design, “launching a worker” is de-coupled from “assigning a work item” in a work distributor, and new handshake mechanisms between a worker and the work-distributor is provided for work assignment, in order to provide persistent kernel functionality. In example embodiments, software specifies the work that has to be done, hardware selects a variable number of workers based on available resources, and a hardware scheduler handshaking with the executing workers assigns more work as previously assigned work is completed and/or more resources become available.