Pipelined Compute Dispatch Queues for Lower Kernel Execution Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Processing requests in graphics processing units can become a bottleneck, preventing the requested work from being performed as quickly as possible.
Innovation Solution
Implementing a work dispatch descriptor queue that allows the work dispatch controller to operate at a different rate than the command queue processor, reducing latency by enabling the command queue processor to work ahead and store processing results, thereby improving throughput.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If the command queue processor directly configures hardware to spawn workgroups, then the processing flow is simple, but the execution latency increases and throughput decreases
Solution Approach 1:
The command queue processor performs preliminary processing of command queue entries and stores the results in a work dispatch descriptor queue before the work dispatch controller needs them. This allows the command queue processor to work ahead and prepare work dispatch descriptors in advance, reducing the execution latency when the work dispatch controller needs to spawn workgroups.
Solution Approach 2:
The processing pipeline is segmented into two independent stages: the command queue processor that generates work dispatch descriptors, and the work dispatch controller that consumes them to spawn workgroups. By introducing the work dispatch descriptor queue as a buffer between these stages, each component can operate at its own optimal speed without blocking the other, thus reducing overall execution latency.
2Productivity
If the command queue processor waits for the work dispatch controller to be ready, then the system remains synchronized, but the throughput decreases
Solution Approach 1:
The work dispatch descriptor queue acts as an intermediary buffer between the command queue processor and the work dispatch controller. This allows the command queue processor to continue processing commands and filling the queue without waiting for the work dispatch controller to be ready, thereby increasing throughput while maintaining system coordination through the queue mechanism.
Solution Approach 2:
The pipeline architecture becomes dynamic where the command queue processor and work dispatch controller can operate at different rates. The work dispatch descriptor queue adapts to the varying processing speeds of the two components, allowing the command queue processor to work ahead when the controller is busy, thus maximizing overall productivity.
Data Source
AI summary
A technique is provided for improving throughput and latency for processing command queue entries that describe work to be performed for compute kernels. The technique includes processing the command queue entries and, instead of directly configuring hardware that spawns workgroups for compute kernel execution, storing work dispatch descriptor entries that describe how to spawn the workgroups. These work dispatch descriptor entries allow a work dispatch controller that spawns the workgroups to work at a different rate than the command queue processor which processes the command queue entries, which helps to reduce latency of execution.