Systolic Matrix Block Streaming for Low-Latency GEMM Acceleration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional hardware accelerators for generalized matrix multiplication (GEMM) operations suffer from high latency due to memory reading and writing patterns, leading to inefficient utilization of matrix multiplication circuits during read and write operations.
Innovation Solution
A hardware accelerator with a systolic array architecture that performs GEMM operations on quantized matrices, applying scale factors to blocks or superblocks, and supports matrix block streaming to reduce latency by iteratively processing input matrix shards without waiting for the entire matrix to be loaded.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional hardware accelerators use standard memory reading and writing patterns for GEMM operations, then the implementation is simple and straightforward, but the latency is high and circuit utilization is inefficient
Solution Approach 1:
The patent divides the input matrices into blocks and processes them in a block-based manner through the systolic array. This segmentation allows continuous processing of matrix blocks without waiting for complete matrix loading, thereby reducing latency and improving circuit utilization efficiency.
Solution Approach 2:
The systolic array is pre-configured with the necessary computational units and data flow paths before operation. Input blocks are loaded and positioned in advance, allowing multiplication operations to begin immediately without waiting for complete data preparation, thus reducing overall operation latency.
2Reliability
If the hardware accelerator waits for the entire input matrix to be loaded before processing, then data accuracy is complete, but the processing latency increases
Solution Approach 1:
The systolic array maintains continuous computational action by processing complete matrix blocks as they become available. Rather than waiting for entire matrices, the architecture continuously multiplies and accumulates block data, ensuring useful action continues without idle periods while maintaining data integrity through proper block-based computation.
Solution Approach 2:
By segmenting matrices into processable blocks, the system can complete and output results for each block independently. This ensures data completeness for each computed block while avoiding the need to wait for entire large matrices, thus reducing latency without sacrificing computational accuracy.
3Measurement precision
If the systolic array processes entire matrices at once, then computational accuracy is maximized, but the time required for processing increases
Solution Approach 1:
The patent segments large matrix multiplication tasks into smaller block-based computations that can be processed independently through the systolic array. Each block computation maintains full numerical accuracy while completing in shorter time, and results are accumulated to produce the final accurate product matrix, thus reducing processing duration without compromising computational precision.
Solution Approach 2:
The systolic array performs preliminary block-based multiplications and accumulations in advance, producing intermediate results that are combined to form the final product. This preliminary processing of blocks allows the complete computation to finish faster while maintaining accuracy through proper accumulation of intermediate results.
Data Source
Figure 1
Figure 2
Figure 3A~3B
AI summary
A hardware accelerator (20) including tiles (24) arranged in a systolic array (22). At each of the tiles, the systolic array receives a first input block (52) that includes first input matrix elements (51) of a first input matrix (50). In each of a plurality of multiplication iterations (72), at each of the tiles, the systolic array receives a respective second input block (62). The systolic array computes tile products (80) of the first input matrix elements and second input matrix elements (61) included in the second input blocks. The systolic array adds the tile products to column-wise partial sums (82) and transmits the column-wise partial sums to subsequent tiles along accumulator rings (84) included in array columns (29) of the systolic array. In a subset (74) of the multiplication iterations, the systolic array outputs product block rows (93) of a product matrix (90). The product block rows each include product matrix blocks (92) computed as rows of the column-wise partial sums.