Memory Aware Scheduling and Cache Access Re-execution for GPU Saturation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern processors, such as GPUs, face performance bottlenecks due to memory intensive workloads saturating memory subsystem resources, leading to serialized memory accesses and inadequate computation to hide memory latency, especially when uncoalesced and irregular memory access patterns cause cache thrashing.
Innovation Solution
The Memory Aware Scheduling and Cache Access Re-execution (MASCAR) approach prioritizes memory requests from one warp over others when the memory subsystem is saturated, using a re-execution queue to manage stalled requests and exploit cache locality, thereby improving the overlap of computation and memory accesses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If memory intensive workloads issue multiple memory requests to DRAM, then data gathering for computation is improved, but memory subsystem resources become saturated causing serialization of memory accesses
Solution Approach 1:
The system dynamically adjusts scheduling behavior based on memory subsystem saturation detection. When saturation is detected, the scheduler switches from allowing multiple warps to issue memory requests simultaneously to serializing memory requests from a single warp, thereby adapting to changing resource availability and preventing further saturation.
Solution Approach 2:
The system periodically monitors memory subsystem saturation and adjusts scheduling policies accordingly. The saturation detection mechanism continuously checks resource availability and triggers scheduling algorithm changes when thresholds are exceeded, creating a periodic feedback loop that maintains optimal performance under varying workload conditions.
2Adaptability or versatility
If uncoalesced and irregular memory access patterns are allowed, then program flexibility is improved, but cache thrashing increases forcing more DRAM requests and worsening memory access serialization
Solution Approach 1:
The system applies different scheduling treatments to different warps based on their access patterns and the current memory subsystem state. When saturation occurs, priority is given to warps with coalesced access patterns that can be served more efficiently, while irregular access patterns are serialized or delayed, thereby optimizing local resource utilization without completely restricting access pattern diversity.
3Productivity
If computation is increased to hide memory latency, then throughput is improved, but the memory subsystem saturation prevents sufficient computation from being initiated
Solution Approach 1:
The system prepares computation by issuing all necessary memory requests from a single warp before allowing that warp to begin computation. This preliminary action ensures that when computation starts, all required data is already in the memory subsystem buffers, eliminating computation idle time caused by waiting for memory requests to complete.
Data Source
AI summary
A multithreaded data processing system performs processing using resource circuitry which is a finite resource. A saturation signal is generated to indicate when the resource circuitry is no longer able to perform processing operations issued to it. This saturations signal may be used to select a scheduling algorithm to be used for further scheduling, such as switching to scheduling from a single thread as opposed to round-robin scheduling from all of the threads. Re-execution queue circuitry is used to queue processing operations which have been enabled to be issued so as to permit other processing operations which may not be blocked by the lack of use of circuitry to attempt issue.