Cooperative Group Arrays for Distributed Shared Memory and SM Synchronization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing GPU architectures face challenges in achieving strong scaling due to limitations in data bandwidth and inefficient data sharing across processing cores, leading to performance bottlenecks and reduced efficiency in parallel processing.
Innovation Solution
Introduce a new programming and hardware model that enables distributed shared memory (DSMEM) across multiple Streaming Multiprocessors (SMs) for direct data sharing and synchronization, allowing concurrent execution and cooperation beyond the single SM level, with hardware-based concurrency guarantees.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data sharing across multiple SMs is implemented using existing GPU architectures, then parallel processing capability is improved, but data bandwidth limitations and synchronization efficiency deteriorate
Solution Approach 1:
The patent merges multiple SMs into a unified cooperative group that shares a common address space and memory resources. This allows threads from different SMs to access the same memory locations directly, eliminating the need for expensive inter-SM data transfer mechanisms and improving data bandwidth efficiency while maintaining parallel processing capability.
Solution Approach 2:
The patent creates a universal memory space that is accessible by all threads within the cooperative group regardless of which SM they are executing on. This multi-functional memory system serves both as local SM memory and as inter-SM communication medium, improving data sharing efficiency without sacrificing parallel processing power.
2Productivity
If distributed shared memory is implemented across multiple SMs, then data sharing efficiency is improved, but hardware complexity increases
Solution Approach 1:
The patent introduces a cooperative group manager as an intermediary layer that handles memory management and synchronization tasks for distributed shared memory across multiple SMs. This mediator abstracts the complexity of inter-SM communication, providing simple load/store instructions to programmers while managing the underlying hardware complexity internally.
Solution Approach 2:
The patent implements self-service mechanisms where the hardware automatically handles memory coherence, synchronization, and address translation for distributed shared memory accesses. The system self-manages the complexity of coordinating multiple SMs through automatic barrier synchronization and memory consistency protocols, reducing the burden on software while improving data sharing efficiency.
3Productivity
If concurrent execution across multiple SMs is enabled, then compute throughput is improved, but synchronization latency increases
Solution Approach 1:
The patent enables continuous useful action by allowing threads in a cooperative group to execute concurrently across multiple SMs while maintaining continuous access to shared memory. The unified address space eliminates idle synchronization wait states, as threads can continuously perform useful work while naturally synchronizing through shared memory operations, improving both throughput and reducing effective synchronization latency.
Data Source
AI summary
A new level(s) of hierarchy—Cooperate Group Arrays (CGAs)—and an associated new hardware-based work distribution/execution model is described. A CGA is a grid of thread blocks (also referred to as cooperative thread arrays (CTAs)). CGAs provide co-scheduling, e.g., control over where CTAs are placed/executed in a processor (such as a GPU), relative to the memory required by an application and relative to each other. Hardware support for such CGAs guarantees concurrency and enables applications to see more data locality, reduced latency, and better synchronization between all the threads in tightly cooperating collections of CTAs programmably distributed across different (e.g., hierarchical) hardware domains or partitions.