NUMA Memory Access Priority and Cache Bypass for Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In non-uniform memory access (NUMA) architectures, memory divergence occurs due to varying latencies between local and remote memory access, leading to stalling of wavefronts as threads wait for remote memory access to complete, even if local memory access is finished, which hampers performance.
Innovation Solution
The system modifies cache behavior by prioritizing and selectively bypassing cache access based on whether memory access requests are for local or remote memory, using priority categories and time-batching to manage queues, and biases cache insertion and eviction policies to reduce latency disparities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If threads access remote memory in a NUMA architecture, then memory access flexibility and data availability are improved, but memory access latency increases significantly
Solution Approach 1:
The patent segments memory access requests into different categories (local memory accesses, remote memory accesses, cache hits, cache misses) and processes them through separate queues with different priority levels. This segmentation allows the system to handle different access types independently, preventing remote memory accesses from blocking local memory accesses, thus resolving the latency flexibility tradeoff.
Solution Approach 2:
The patent implements dynamic priority assignment where the priority of memory access requests is not fixed but dynamically adjusted based on whether they are local or remote accesses, and whether they hit or miss the cache. This dynamic prioritization allows the system to adapt to varying access patterns and minimize overall latency while maintaining access flexibility.
2Reliability
If the system waits for all threads to complete memory access before proceeding, then data consistency is ensured, but wavefront stalling occurs reducing productivity
Solution Approach 1:
The patent divides the wavefront execution into segments based on memory access completion status. Threads are categorized into those that have completed memory access and those that are still waiting. The system can proceed with threads that have completed their access while maintaining data consistency through proper synchronization mechanisms, thus avoiding complete wavefront stalling.
Solution Approach 2:
The patent performs preliminary categorization of threads based on their memory access status before the wavefront needs to proceed. By pre-identifying which threads have completed memory access and which are still waiting, the system can plan the execution schedule in advance, allowing productive work to continue while maintaining data consistency through controlled synchronization points.
3Quantity of substance
If cache access is performed for all memory requests, then data availability is improved, but cache bandwidth is overloaded by remote memory requests
Solution Approach 1:
The patent segments cache access into two distinct paths: one for local memory requests and another for remote memory requests. Local requests go through the cache as before, while remote requests are handled through a separate mechanism that does not overloads the cache bandwidth. This segmentation allows the cache to serve its primary function without being overwhelmed by remote access traffic.
Solution Approach 2:
The patent extracts remote memory access handling from the traditional cache access path. By separating remote requests from the cache bandwidth path, the system prevents remote memory requests from consuming cache bandwidth that would otherwise be available for local memory requests and cache hits, thus optimizing overall data availability efficiency.
Data Source
AI summary
A miss in a cache by a thread in a wavefront is detected. The wavefront includes a plurality of threads that are executing a memory access request concurrently on a corresponding plurality of processor cores. A priority is assigned to the thread based on whether the memory access request is addressed to a local memory or a remote memory. The memory access request for the thread is performed based on the priority. In some cases, the cache is selectively bypassed depending on whether the memory access request is addressed to the local or remote memory. A cache block is requested in response to the miss. The cache block is biased towards a least recently used position in response to requesting the cache block from the local memory and towards a most recently used position in response to requesting the cache block from the remote memory.