Gang Preemption for GPU Workload Switching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing preemption schemes in graphics processing units (GPUs) introduce delays and can negatively affect dependent workloads due to the lack of consideration for interdependencies between workloads executing on different pipelines, leading to inconsistent state resumption.
Innovation Solution
The 'gang preemption' technique allows for low-latency switching between workloads by suspending and resuming dependent workloads on GPUs without draining pipelines, using a suspend memory to store state information and ensuring precise state restoration across multiple pipelines, including persistent storage for power management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If existing preemption schemes are used to switch between workloads, then workload switching can be performed, but delays are introduced and processing efficiency deteriorates
Solution Approach 1:
The system performs preliminary actions by pre-calculating and storing state information of workloads in suspend memory before preemption occurs. This allows the GPU to rapidly restore workload state without performing lengthy save operations during the actual preemption event, thereby reducing preemption delay while maintaining switching capability.
2Reliability
If pipelines are drained before preemption to save state information, then state can be preserved, but processing time increases and productivity decreases
Solution Approach 1:
The invention extracts only the essential state information needed for workload resumption from the pipeline and stores it in suspend memory, rather than draining the entire pipeline. This selective extraction maintains state preservation accuracy while avoiding the productivity loss associated with complete pipeline draining.
3Device complexity
If preemption does not consider interdependencies between workloads, then switching is simpler, but dependent workloads are negatively affected and reliability decreases
Solution Approach 1:
The system implements feedback mechanisms that monitor dependencies between workloads executing on different pipelines. When a preemption event occurs, the system checks for dependent workloads and adjusts the preemption timing or state saving strategy accordingly, ensuring that dependent workloads are not adversely affected while maintaining manageable system complexity.
4Quantity of substance
If state information is stored in external off-chip memory, then more state can be saved, but access time increases and latency increases
Solution Approach 1:
The invention implements a nested memory structure where suspend memory is integrated within or closely coupled to the GPU chip, forming an intermediate layer between on-chip resources and external off-chip memory. This nested arrangement provides sufficient storage capacity for state information while maintaining fast access times, thereby reducing latency compared to purely external storage solutions.
Data Source
AI summary
A first workload is executed in a first subset of pipelines of a processing unit. A second workload is executed in a second subset of the pipelines of the processing unit. The second workload is dependent upon the first workload. The first and second workloads are suspended and state information for the first and second workloads is stored in a first memory in response to suspending the first and second workloads. In some cases, a third workload executes in a third subset of the pipelines of the processing unit concurrently with executing the first and second workloads. In some cases, a fourth workload is executed in the first and second pipelines after suspending the first and second workloads. The first and second pipelines are resumed on the basis of the stored state information in response to completion or suspension of the fourth workload.