Distributed Shared Memory for GPU Strong Scaling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current GPU architectures face challenges in achieving strong scaling due to limitations in data bandwidth and memory access efficiency, particularly when dealing with massive parallel computations required for deep learning and high performance computing applications.
Innovation Solution
The introduction of a new programming and hardware execution model that enables distributed shared memory (DSMEM) across multiple streaming multiprocessors (SMs), allowing for direct data sharing and synchronization between groups of processing cores, thereby overcoming traditional memory access bottlenecks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional shared memory is used within a single streaming multiprocessor, then thread communication is efficient, but data bandwidth and memory access efficiency are insufficient for massive parallel computations across multiple SMs
Solution Approach 1:
The patent segments the shared memory into distributed shared memory blocks located at different SMs, allowing each SM to access memory across the entire distributed memory space rather than being confined to local memory only. This segmentation enables scalable memory access patterns that support massive parallel computations while maintaining efficient data sharing.
Solution Approach 2:
The patent introduces an intermediary memory management mechanism that coordinates access to distributed shared memory across multiple SMs. This intermediary layer handles memory allocation, access routing, and synchronization, thereby simplifying the complexity of memory access while enabling efficient data bandwidth utilization across the distributed memory architecture.
2Productivity
If the number of processing cores is increased to achieve strong scaling, then computational capability improves, but memory bandwidth becomes a bottleneck
Solution Approach 1:
The patent transitions from a single-dimension memory access model (local memory only) to a multi-dimension memory access model that includes distributed shared memory across multiple SMs. This dimensional expansion allows memory bandwidth to scale with the number of processing cores by providing additional memory pathways, thereby supporting strong scaling without being bottlenecked by traditional memory bandwidth limitations.
3Ease of operation
If data is shared through global memory, then access is simplified, but memory latency increases significantly
Solution Approach 1:
The patent implements a nested memory hierarchy where distributed shared memory blocks are embedded within the broader memory system. This nested structure allows for fast local access within each SM while maintaining the ability to access memory across SMs through the distributed shared memory mechanism, thereby achieving both simplicity and high speed without the latency penalties of traditional global memory access.
Data Source
AI summary
Distributed shared memory (DSMEM) comprises blocks of memory that are distributed or scattered across a processor (such as a GPU). Threads executing on a processing core local to one memory block are able to access a memory block local to a different processing core. In one embodiment, shared access to these DSMEM allocations distributed across a collection of processing cores is implemented by communications between the processing cores. Such distributed shared memory provides very low latency memory access for processing cores located in proximity to the memory blocks, and also provides a way for more distant processing cores to also access the memory blocks in a manner and using interconnects that do not interfere with the processing cores' access to main or global memory such as backed by an L2 cache. Such distributed shared memory supports cooperative parallelism and strong scaling across multiple processing cores by permitting data sharing and communications previously possible only within the same processing core.