Software Pre-execution Loop for Irregular Memory Access Bandwidth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High bandwidth memories (HBMs) have shifted the performance bottleneck from memory bandwidth to memory latency, particularly for applications with irregular memory accesses, where existing hardware-based solutions like hardware prefetchers are ineffective, leading to poor bandwidth utilization.
Innovation Solution
A software-based approach that modifies existing code to split loops into pre-execution and modified loops, with the pre-execution loop handling memory accesses and address generation to prefetch data for the modified loop, leveraging the greater number of L2 MSHRs to reduce stalls and improve bandwidth utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If hardware prefetchers are used to mitigate memory latency stalls, then memory latency is reduced for regular memory accesses, but bandwidth utilization remains poor for irregular memory accesses
Solution Approach 1:
The patent applies preliminary action by executing a pre-execution loop that performs memory accesses and address generation in advance of the main computational loop. This prefetches data into the cache hierarchy before it is needed, thereby hiding memory latency without requiring hardware prefetchers. The pre-execution loop calculates addresses and loads data preliminarily, so when the modified loop executes, data is already available in cache, improving both latency and bandwidth utilization for irregular memory access patterns.
2Productivity
If software prefetching is implemented to improve bandwidth utilization, then memory bandwidth is better utilized, but vectorization is broken and significant overhead is incurred
Solution Approach 1:
The patent applies segmentation by dividing the original loop into two distinct loops: a pre-execution loop that handles memory accesses and address generation, and a modified loop that performs computation. This segmentation allows the memory-intensive operations to be separated from computation, enabling the pre-execution loop to be optimized for memory access patterns while the modified loop maintains vectorization for computational efficiency. The segmentation eliminates the need for complex software prefetching instructions within the computational loop, reducing code complexity while improving bandwidth utilization.
3Reliability
If L1 MSHRs are used for memory access management, then memory access control is provided, but the limited number of L1 MSHRs cannot fully utilize HBM bandwidth
Solution Approach 1:
The patent applies dimensionality change by utilizing L2 MSHRs as an additional dimension for memory access management beyond the traditional L1 MSHRs. The pre-execution loop is designed to generate enough outstanding memory requests to fully utilize the larger L2 MSHR resources, thereby accessing more of the HBM bandwidth. This dimensional expansion from L1 to L2 cache hierarchy allows the system to overcome the bottleneck of limited L1 MSHRs and fully exploit the high bandwidth of HBM memory.
Data Source
AI summary
Systems and methods are configured to receive code containing an original loop that includes irregular memory accesses. The original loop can be split. A pre-execution loop that contains code to prefetch content of the memory can be generated. Execution of the pre-execution loop can access memory inclusively between a starting location and the starting location plus a prefetch distance. A modified loop that can perform at least one computation based on the content prefetched with execution of the pre-execution loop can be generated. Execution of the main loop can to follow the execution of the pre-execution loop. The original loop can be replaced with the pre-execution loop and the modified loop.