Local Memory Sharing Between GPU Kernels
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current graphics processing units (GPUs) face limitations in efficiently processing graphics and machine-learning operations due to the complexity of parallel processing techniques and the need for optimized architectures that can handle diverse computational workloads.
Innovation Solution
A GPU architecture is developed that incorporates a parallel processing unit with a scheduler, memory crossbar, and shared memory management, enabling efficient execution of interdependent thread groups and local memory sharing between kernels, thereby optimizing graphics and machine-learning operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If local memory is shared between kernels to improve processing efficiency, then memory access speed and parallel processing capability are improved, but memory management complexity and potential data conflicts increase
Solution Approach 1:
The shared local memory is divided into multiple banks (e.g., bank 0, bank 1, bank 2) that can be independently accessed by different thread groups or kernels. This segmentation allows parallel memory access operations to proceed simultaneously without conflicts, maintaining high processing efficiency while simplifying the management of each individual memory bank.
Solution Approach 2:
A memory management unit (MMU) or controller acts as an intermediary between multiple kernels/thread groups and the shared local memory banks. This intermediary coordinates access requests, handles address translation, and manages data coherence, thereby enabling efficient memory sharing while abstracting the complexity from the kernels themselves.
2Productivity
If interdependent thread groups are executed concurrently to maximize parallel processing, then throughput is improved, but scheduling complexity and execution coordination increase
Solution Approach 1:
Thread groups are organized into waves or bundles before execution, with their dependencies and data flow pre-analyzed and structured. This preliminary organization allows the scheduler to issue pre-coordinated batches of thread groups to the execution units, enabling concurrent execution while maintaining proper dependency ordering without complex real-time scheduling decisions.
Solution Approach 2:
The execution architecture employs dynamic wave scheduling where thread groups are issued in flexible waves that can be adjusted based on readiness and dependency status. Execution units can dynamically start, pause, and resume different thread group waves, allowing concurrent execution of interdependent thread groups while adapting to changing execution states and data availability.
Data Source
AI summary
One embodiment provides for a general-purpose graphics processing unit comprising a set of processing elements to execute one or more thread groups of a second kernel to be executed by the general-purpose graphics processor, an on-chip memory coupled to the set of processing elements, and a scheduler coupled with the set of processing elements, the scheduler to schedule the thread groups of the kernel to the set of processing elements, wherein the scheduler is to schedule a thread group of the second kernel to execute subsequent to a thread group of a first kernel, the thread group of the second kernel configured to access a region of the on-chip memory that contains data written by the thread group of the first kernel in response to a determination that the second kernel is dependent upon the first kernel.