GEMM Tiling With Early Memory Access for Lower Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for Generalized Matrix Multiplication (GEMM) on parallel processors, such as GPUs, face challenges with scalability and efficiency, particularly with large-scale operations and matrices exhibiting sparsity or high dimensionality, leading to memory access latency and underutilization of throughput capabilities.
Innovation Solution
Implementing early memory access techniques that overlap memory accesses within iterations of GEMM kernels to maximize L2 cache capacity and memory throughput, reducing overall kernel latency and thread stall cycles without requiring additional resources or hardware modifications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional matrix multiplication methods are used on parallel processors, then basic computational tasks can be performed, but memory access latency increases and throughput capabilities are underutilized for large-scale operations
Solution Approach 1:
The patent applies preliminary action by loading and storing matrix data to L2 cache before the main computation loop executes. Specifically, input matrices A and B are loaded into L2 cache prior to processing, and intermediate results are stored in L2 cache during computation. This pre-positioning of data in faster cache memory reduces memory access latency during the computationally intensive loops, directly addressing the contradiction between throughput and latency.
Solution Approach 2:
The patent segments the computation into distinct phases with specialized memory access patterns. The GEMM computation is divided into loop iterations that process different portions of the matrices, with each iteration accessing specific cache lines. This segmentation allows the system to overlap memory accesses across different iterations and utilize L2 cache capacity more effectively, improving both throughput and reducing latency.
2Quantity of substance
If existing GEMM methods are applied to large-scale matrices, then comprehensive data processing is achieved, but scalability and efficiency deteriorate due to memory bandwidth limitations
Solution Approach 1:
The patent exploits the temporal dimension by overlapping memory accesses across different loop iterations. Instead of sequentially completing one iteration before starting the next, the method initiates memory loads in iteration i while computations from iteration i-1 are still in progress. This temporal dimension exploitation allows the system to hide memory latency and maintain high processing efficiency even with large matrix volumes.
Solution Approach 2:
The patent changes the timing parameters of memory accesses by performing loads and stores at different points in the computation loop than traditional methods. Specifically, input data is loaded earlier and intermediate results are stored at optimized points in the iteration, changing the temporal parameters of memory operations to maximize bandwidth utilization and processing efficiency for large datasets.
3Ease of manufacture
If conventional memory access patterns are used in GEMM kernels, then simple implementation is maintained, but L2 cache capacity and memory bandwidth are underutilized
Solution Approach 1:
The patent ensures continuity of useful action by maintaining overlapping memory operations throughout the computation. Memory loads and stores are continuously initiated across loop iterations without idle cycles, keeping the memory subsystem continuously utilized. This continuous operation maximizes L2 cache capacity utilization and memory bandwidth throughput while maintaining a relatively simple implementation structure.
Data Source
AI summary
A processor includes a plurality of processing elements. Each processing element is configured to obtain a first plurality of submatrices from a first input matrix and a second plurality of submatrices from a second input matrix. The first and second plurality of submatrices, for at least a first iteration of a plurality of matrix multiply iterations, each include at least one submatrix that is distinct from submatrices obtained by the other processing elements. The processing element performs one or more matrix multiplication operations on the first plurality of submatrices and the second plurality of submatrices to generate partial results for an output submatrix of an output matrix associated with the processing element. The processing element generates a portion of the output matrix by combining the partial results in the memory for the output submatrix. The output submatrices generated by each of the processing elements form the output matrix.