Prefetch Kernels Reduce Memory Latency on Data-Parallel Processors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data-parallel processor architectures face high hardware costs and power consumption due to large on-chip storage requirements for hiding memory access latencies, which become impractical as the number of concurrent threads increases and off-chip memory latencies grow.
Innovation Solution
Implementing a prefetch kernel that runs ahead of the compute or graphics kernel to warm up Translation Lookaside Buffers and caches, reducing memory access latencies and intermediate state storage requirements by performing memory operations based on the corresponding processing kernel, thereby minimizing hardware costs and power consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If a large number of concurrently live wavefronts are used to hide memory access latencies, then memory access latencies are hidden, but hardware costs and power consumption increase due to large register files and on-chip memories
Solution Approach 1:
The patent applies preliminary action by executing a prefetch kernel before the main compute kernel to pre-load data into caches and warm up Translation Lookaside Buffers. This advance preparation reduces the memory access latency for the main kernel without requiring a large number of concurrently live wavefronts, thereby hiding latency through time-separated prefetching rather than spatial parallelism.
2Loss of time
If a large number of concurrently live wavefronts are used to hide memory access latencies, then memory access latencies are hidden, but power consumption increases
Solution Approach 1:
The prefetch kernel performs data preparation and cache warming in advance, allowing the main compute kernel to execute with reduced memory access latency. This time-separated approach avoids the need to maintain many concurrently live wavefronts, thereby reducing power consumption while still achieving latency hiding through the prefetch-compute pipeline.
3Loss of time
If the number of concurrent threads is increased, then memory access latencies are better hidden, but intermediate state storage requirements increase
Solution Approach 1:
The prefetch kernel executes memory operations in advance of the compute kernel, warming up caches and TLBs before the main computation begins. This preliminary data preparation reduces the intermediate state storage requirements for the compute kernel by ensuring data is already in fast memory, eliminating the need to maintain large amounts of intermediate state for concurrent threads.
Data Source
AI summary
Embodiments include methods, systems and non-transitory computer-readable computer readable media including instructions for executing a prefetch kernel that includes memory accesses for prefetching data for a processing kernel into a memory, and, subsequent to executing at least a portion of the prefetch kernel, executing the processing kernel where the processing kernel includes accesses to data that is stored into the memory resulting from execution of the prefetch kernel.