GPU Shuffle Accelerator With Phased Crossbar Data Exchange
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing graphics processing units (GPUs) face inefficiencies in exchanging data between instances within a shader core slot, particularly due to high latency and low bandwidth in shared memory and the inefficiency of synchronization methods like barriers and atomics, leading to unnecessary idling of instances.
Innovation Solution
The implementation of a shuffle accelerator with routing, slave, and master logic to efficiently shuffle data between instances in a shader core by dividing the data exchange into multiple phases, using a smaller crossbar-like hardware structure to connect subsets of instances.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data exchange between instances uses shared memory with barriers and atomics, then data can be exchanged between instances, but latency is high and bandwidth is low causing instances to idle
Solution Approach 1:
The shuffle accelerator divides data exchange into multiple phases, with each phase handling a subset of instances. This segmentation allows parallel processing of data exchanges across different instance subsets, reducing overall latency and eliminating idle time while maintaining reliable data exchange between all instances.
2Productivity
If a large crossbar is used to connect all instances for data exchange, then data exchange bandwidth is high, but hardware cost increases
Solution Approach 1:
Instead of implementing a single large crossbar connecting all instances, the system segments the crossbar into multiple smaller crossbars, each handling a subset of instances during specific phases. This segmentation maintains high data exchange bandwidth through parallel operations while significantly reducing hardware complexity and cost compared to a full N×N crossbar.
3Adaptability or versatility
If all instances participate in data exchange simultaneously, then complete data shuffling is achieved, but synchronization overhead increases causing idling
Solution Approach 1:
The system segments the instance population into multiple subsets that participate in data exchange during different phases. Each phase synchronizes only the required subset of instances, eliminating unnecessary synchronization overhead and idle time for instances not involved in the current phase, while still achieving complete data shuffling across all instances through the sequence of phases.
4Reliability
If global or local memory is used for data exchange, then data can be shared between instances, but bandwidth is low and latency is high
Solution Approach 1:
The shuffle accelerator introduces a dedicated intermediary hardware structure (the phased crossbar network) that mediates data exchange between instances. This intermediary provides high-bandwidth, low-latency direct connections for data shuffling operations, eliminating the bottleneck of using general-purpose global or local memory while maintaining reliable data sharing capability between all instances.
Data Source
AI summary
Shuffle accelerators for shuffling data on a shader core of a graphics processing unit include routing logic, slave logic and master logic. The routing logic selectively connects data input ports to a plurality of data output ports. The slave logic selectively provides data from a first set of instances to the plurality of data input ports and receives data from the plurality of data output ports for a second set of instances. The master logic is configured to, in response to receiving a shuffle instruction that identifies a shuffle of data between the plurality of instances, cause the routing logic and the slave logic to perform the identified shuffle of data in a plurality of phases, wherein in each phase of the plurality of phases a subset of the instances of the plurality of instances receive data from a subset of the instances of the plurality of instances.