GPU Shuffle Accelerator for Low-Latency Shader Data Exchange
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing graphics processing units (GPUs) face inefficiencies in exchanging data between instances within a shader core slot, particularly due to high latency and low bandwidth in shared memory and synchronization methods like barriers and atomics, leading to unnecessary idling of instances.
Innovation Solution
A shuffle accelerator is introduced, comprising routing, slave, and master logic to efficiently shuffle data between instances in phases, using dedicated hardware to connect subsets of instances via a smaller crossbar-like structure, reducing the need for large crossbars and minimizing latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If shared memory and synchronization methods (barriers, atomics) are used for data exchange between instances, then data exchange functionality is provided, but latency is high and bandwidth is low causing instances to idle
Solution Approach 1:
The patent introduces a shuffle accelerator as an intermediary component between instances and shared memory. This accelerator includes routing logic with input ports, output ports, and switching hardware that directly routes data between instances, bypassing the slow shared memory path. The master logic coordinates shuffles by dividing them into phases, while slave logic handles individual instance data movements, creating a dedicated data exchange pathway that reduces latency and eliminates idle time.
2Adaptability or versatility
If a large crossbar is used to connect all instances for data shuffling, then complete data exchange between any instances is enabled, but device complexity and hardware resources increase
Solution Approach 1:
The patent segments the shuffle operation into multiple phases, where each phase handles a subset of instances. The routing logic is divided into master logic that controls the overall shuffle process and slave logic that handles individual instance data movements. This segmentation allows the system to achieve complete data exchange capability without requiring a single large crossbar connecting all instances simultaneously, thereby reducing hardware complexity while maintaining versatility.
3Reliability
If instances exchange data via shared memory, then data sharing is achieved, but bandwidth is low and performance is reduced
Solution Approach 1:
The patent replaces the mechanical shared memory access mechanism with a dedicated routing system. Instead of instances writing to and reading from shared memory locations, the routing logic directly connects instance output ports to input ports through switching hardware. This substitution of the data exchange mechanism eliminates the bandwidth limitations of shared memory while maintaining the data sharing capability, thereby significantly improving productivity.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Shuffle accelerators for shuffling data between a plurality of instances executing a shader on a shader core of a graphics processing unit. The shuffle accelerators include routing logic, slave logic and master logic. The routing logic comprises a plurality of data input ports, a plurality of data output ports, and hardware to selectively connect one or more of the plurality of data input ports to one or more of the plurality of data output ports. The slave logic is configured to selectively provide data from a first set of instances to one or more of the plurality of data input ports and receive data from one or more of the plurality of data output ports for a second set of instances. The master logic is configured to, in response to receiving a shuffle instruction that identifies a shuffle of data between the plurality of instances, cause the routing logic and the slave logic to perform the identified shuffle of data in a plurality of phases, wherein in each phase of the plurality of phases a subset of the instances of the plurality of instances receive data from a subset of the instances of the plurality of instances.