Cooperative Work-Stealing Scheduler for CPU-GPU Task Offloading
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional GPU scheduling policies limit the flexibility of task scheduling and resource utilization due to hardware-defined scheduling and the bulk-synchronous programming model, leading to underutilization of GPU resources, especially in applications requiring multi-processor synchronization and task locality.
Innovation Solution
A heterogeneous, cooperative work-stealing scheduler that offloads tasks from a CPU to a GPU when a sequential cutoff is reached, utilizing a shared memory system with doorbell and task registers to manage task execution and synchronization across CPU and GPU cores.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional centralized task scheduling is used, then task distribution can be controlled centrally, but the system cannot scale efficiently and creates a bottleneck
Solution Approach 1:
The centralized scheduler is segmented into distributed scheduler components across multiple compute nodes. Each node runs its own scheduler that can independently make scheduling decisions, eliminating the single-point bottleneck while maintaining coordinated task distribution across the cluster.
Solution Approach 2:
The scheduling architecture transitions from a single-dimensional centralized model to a multi-dimensional distributed model, adding the dimension of node autonomy. This allows parallel scheduling operations across multiple dimensions (nodes) simultaneously, improving scalability and throughput.
2Productivity
If work is stolen from other compute nodes, then idle compute nodes can be utilized, but cache coherency traffic increases reducing efficiency
Solution Approach 1:
The system implements local data ownership where each compute node maintains ownership of data blocks it processes. When tasks are stolen, the system checks local ownership first and only initiates remote cache line invalidation when necessary, minimizing cross-node cache coherency traffic while maintaining high node utilization.
Solution Approach 2:
Instead of maintaining strict cache coherency for all shared data, the system creates local copies of data blocks at computing nodes. This allows nodes to work with local copies independently, reducing the need for cache coherency protocols and associated traffic between nodes.
3Reliability
If fine-grained data locking is implemented, then data access safety is ensured, but performance degrades due to lock overhead
Solution Approach 1:
The system extracts the locking mechanism from fine-grained individual data access and replaces it with coarse-grained data block ownership. Each compute node owns entire data blocks, eliminating the need for locks on individual data elements within those blocks, thus removing lock overhead while maintaining data safety through ownership boundaries.
Solution Approach 2:
Instead of applying locking to every data access (excessive action), the system applies ownership rules selectively at data block boundaries (partial action). This selective approach ensures data safety where needed while avoiding lock overhead in the majority of local access operations.
4Speed
If data is replicated across compute nodes, then data access speed improves, but memory bandwidth is consumed and data consistency becomes difficult to maintain
Solution Approach 1:
Data blocks are pre-positioned at compute nodes before they are needed for processing. The scheduler proactively distributes data blocks to relevant compute nodes in advance, so when tasks arrive, the data is already locally available, achieving fast access without continuous replication or caching.
Solution Approach 2:
The system discards the need for continuous data replication across all nodes. Instead, data blocks are selectively placed at specific compute nodes based on task requirements, and can be recovered or redistributed as tasks are scheduled, optimizing memory bandwidth usage while maintaining fast local access.
Data Source
Figure 1
Figure 2A~2B
Figure 3A~3B
AI summary
A method for use in a computing system having a central processing unit (CPU) and a graphics processing unit (GPU), the method comprising: assigning a first memory portion and a second memory portion to: a worker thread of a work-stealing scheduler and an execution unit that is part of the GPU; retrieving a task from a queue associated with the worker thread; detecting, by the worker thread, whether a cutoff condition for the task is satisfied; when the cutoff condition is not satisfied, dividing the task into two or more additional tasks and adding the two or more additional tasks to the queue; when the cutoff condition is satisfied, storing first data corresponding to the task in the second memory portion, issuing a memory fence acquire instruction; and storing a first value in the first memory portion.