Streaming Wave Coalescer Circuit Reduces GPU Thread Divergence
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Thread divergence in processing units such as GPUs slows down overall execution as compute units execute diverged threads sequentially rather than in parallel, and this divergence tends to increase over time.
Innovation Solution
A Streaming Wave Coalescer (SWC) circuit reorders and reconstitutes SIMT waves by sorting threads using integer lane key values, merging subgroups with matching sort keys into sort bins, and emitting fully populated bins as reconstituted waves for execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If threads are executed in parallel without reordering, then execution speed is maintained for homogeneous threads, but thread divergence causes sequential execution and slows down overall processing
Solution Approach 1:
The Streaming Wave Coalescer performs preliminary reordering of threads within waves before they reach the compute units. By sorting threads based on sort keys that indicate expected execution paths, the SWC prepares waves in advance so that homogeneous threads are grouped together, enabling parallel execution and avoiding sequential processing due to divergence.
Solution Approach 2:
The SWC changes the ordering parameter of threads within waves by using sort keys to reposition threads. This parameter change transforms the thread arrangement from a potentially divergent order to a homogeneous order, allowing the compute units to execute threads in parallel rather than sequentially.
2Productivity
If threads are reordered to reduce divergence, then execution efficiency improves, but additional circuitry and complexity are introduced into the dispatch pipeline
Solution Approach 1:
The Streaming Wave Coalescer acts as an intermediary component between the instruction cache and compute units. It introduces sort key generation logic and thread reordering mechanisms that mediate the flow of threads, sorting them before execution without requiring fundamental changes to the existing dispatch pipeline architecture.
Solution Approach 2:
The SWC segments the wave processing by dividing threads into subgroups based on sort keys. Each subgroup of threads with matching sort keys is processed separately through sort bins, allowing the system to handle divergence by creating multiple sorted segments rather than processing all threads uniformly.
3Productivity
If sort bins are used to group threads by sort keys, then thread homogeneity increases and divergence reduces, but memory and storage requirements increase
Solution Approach 1:
The sort bins in the Streaming Wave Coalescer are designed to be temporary storage structures that are filled and then emptied in a continuous stream. After threads are sorted and dispatched to compute units, the sort bins are cleared and reused for the next wave, avoiding the need for permanent large-capacity storage while still providing sufficient buffering for reordering operations.
Data Source
AI summary
A Streaming Wave Coalescer (SWC) circuit stores a first set of state values associated with a first subset of threads of a first wave in a bin based on each of the first subset of threads including a first set of instructions to be executed. A second set of state values associated with a second subset of threads of a second wave is stored in the bin based on each of the second subset of threads including the first set of instructions to be executed and based on the first wave and the second wave both being associated with a hard key. A third wave is formed from the threads of the first subset and the second subset and is emitted for execution. As a result of reorganizing the threads and reconstituting a different wave, thread divergence of waves sent for execution is reduced.