GPU Dynamic Pipelining for Multi-Workload Kernel Scheduling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional techniques for workload execution on graphics processing units (GPUs) are inefficient in terms of conserving computing resources, such as thread and cache utilization, and do not effectively manage parallelism and data movement.
Innovation Solution
A mechanism for dynamic pipelining and efficient scheduling of multiple GPGPU workloads on GPUs without requiring additional hardware, utilizing OpenCL 2.0 for parallel scheduling and managing events and criteria to ensure seamless processing, including kernel fusion and cache management to maximize resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional workload execution techniques are used on GPUs, then implementation is simple, but thread and cache utilization are inefficient
Solution Approach 1:
The patent implements dynamic pipelining where the GPU can dynamically switch between processing different workload stages (first workload, second workload, third workload) in a pipeline fashion. The system dynamically adjusts thread allocation and cache management based on the current pipeline stage, allowing efficient resource utilization without static pre-allocation overhead.
Solution Approach 2:
The workload execution is segmented into multiple independent stages (first workload, second workload, third workload) that can be processed simultaneously in parallel pipelines. Each stage can be managed independently with its own thread blocks and cache resources, improving overall throughput without requiring complete resource re-allocation.
2Loss of time
If multiple workloads are processed sequentially, then resource management is simple, but processing time increases and device stalls
Solution Approach 1:
The patent maintains continuous GPU utilization by implementing a pipeline where while the first workload is being processed, the second workload is being prepared, and the third workload is being initialized. This eliminates idle periods and ensures continuous useful action across all workload stages, maximizing processor utilization.
Solution Approach 2:
Subsequent workloads are prepared in advance while previous workloads are still being processed. The second workload can be pre-processed and staged while the first workload executes, and the third workload can be prepared while the second executes, reducing overall processing time through overlapping operations.
3Loss of energy
If kernel fusion is implemented to reduce data movement, then data movement overhead is reduced, but shared memory usage increases
Solution Approach 1:
The patent applies kernel fusion selectively to specific workload stages where data movement benefits are most significant, rather than uniformly across all kernels. By identifying which kernel combinations provide the greatest data movement reduction while maintaining acceptable shared memory usage, the system optimizes the trade-off locally rather than globally.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A mechanism is described for facilitating dynamic pipelining of workload executions at graphics processing units on computing devices. A method of embodiments, as described herein, includes generating a command buffer having a plurality of kernels relating to a plurality of workloads to be executed at a graphics processing unit (GPU), and pipelining the workloads to be processed at the GPU, where pipelining includes scheduling each kernel to be executed on the GPU based on at least one of availability of resource threads and status of one or more dependency events relating to each kernel in relation to other kernels of the plurality of kernels.