GPU Graph-to-Stream Scheduling for Low-Overhead Kernel Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing GPU implementations face inefficiencies in launching and scheduling kernels due to high overheads and suboptimal utilization of parallel processing capabilities, particularly in scenarios with many short-duration kernels and complex dependencies, leading to degraded performance.
Innovation Solution
An adaptive graph-to-stream scheduling method that partitions computation graphs into waves based on kernel dependencies and resource usage, mapping kernels into multiple streams for concurrent execution while maintaining dependencies and balancing load, using techniques like breadth-first search, synchronization kernels, and dynamic remapping.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If kernels are launched one-by-one by the host CPU, then kernel execution can be controlled individually, but launch overhead becomes significant when kernels are many and of short duration
Solution Approach 1:
The patent merges multiple individual kernel launches into a single computation graph launch operation. The computation graph encapsulates multiple kernels and their dependencies, allowing the host CPU to launch them as one unified operation to the GPU, thereby eliminating repeated launch overhead while maintaining individual kernel controllability through the graph structure.
Solution Approach 2:
The patent performs preliminary organization of kernels into a computation graph structure before launching to the GPU. By pre-defining kernel dependencies and execution order in the graph, the system prepares the workload in advance, allowing efficient batch processing without repeated launch overhead during execution.
2Loss of time
If computation graph is launched as a single operation to reduce overhead, then launch overhead is reduced, but mapping kernels to multi-stream becomes complex and critical for GPU utilization
Solution Approach 1:
The patent segments the computation graph into multiple waves based on kernel execution dependencies. Each wave contains kernels that can execute concurrently without violating dependency constraints. This segmentation transforms the complex mapping problem into a structured multi-phase process, where each wave is mapped to available GPU streams independently, simplifying the overall mapping complexity.
Solution Approach 2:
The patent implements dynamic kernel-to-stream mapping that adapts to runtime conditions. The system monitors GPU stream availability and dynamically assigns waves to streams based on current resource state, enabling flexible utilization of multi-stream concurrency while maintaining dependency correctness through the wave structure.
3Productivity
If kernels are mapped to multiple streams for concurrent execution, then GPU utilization efficiency improves, but maintaining kernel dependency correctness becomes challenging
Solution Approach 1:
The patent segments kernels into waves based on dependency levels, where each wave represents a stage of execution. This segmentation ensures that kernels within the same wave have no dependencies on each other and can safely execute concurrently in different streams, while dependencies between waves are maintained through sequential wave execution, thus preserving correctness while enabling concurrency.
Solution Approach 2:
The wave structure acts as an intermediary layer between the computation graph and the GPU streams. It mediates the mapping process by providing a structured interface that naturally enforces dependency constraints while enabling stream-level parallelism, thus reconciling the conflict between concurrency and correctness.
4Reliability
If resource usage is considered when mapping kernels to streams, then resource conflicts are avoided, but mapping complexity increases
Solution Approach 1:
The patent segments the mapping process into two independent phases: wave formation based on dependencies, and stream assignment based on resource usage. This segmentation allows each phase to focus on a single concern, reducing overall mapping complexity while ensuring resource conflicts are avoided through resource-aware stream assignment in the second phase.
Solution Approach 2:
The patent introduces resource usage parameters as additional criteria for stream assignment. By incorporating resource metrics (such as memory usage, compute unit requirements) into the mapping decision, the system dynamically adjusts stream assignment to avoid resource conflicts, transforming a simple dependency-based mapping into a multi-parameter optimization that prevents conflicts without excessive complexity.
Data Source
AI summary
This application describes an accelerator, a computer system, and a method for adaptive graph-to-stream scheduling in processors. An example method may include receiving a computation graph for a GPU, the computation graph comprising (1) a plurality of nodes representing a plurality of kernels for the GPU to execute and (2) a plurality of edges representing execution dependencies among the plurality of kernels; performing one or more wave partitions on the computation graph to determine a plurality of waves of kernels; obtaining a kernel resource table comprising resource usage of each kernel; mapping the plurality of kernels into a plurality of streams based on the plurality of waves and the kernel resource table; and executing the plurality of streams on the GPU, wherein kernels mapped in a same stream are executed sequentially by the GPU, and kernels mapped to different streams are concurrently executable by the GPU.