Systolic Matrix Block Streaming for Low-Latency GEMM Acceleration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional hardware accelerators for generalized matrix multiplication (GEMM) operations suffer from high latency due to memory reading and writing patterns, leading to inefficient utilization of matrix multiplication circuits during read and write operations.
Innovation Solution
A hardware accelerator with a systolic array that processes quantized matrix elements and applies scale factors at the hardware level, performing GEMM operations through matrix block streaming and utilizing a systolic array to compute tile products and accumulate column-wise partial sums efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional hardware accelerators load and write complete matrices to memory, then memory access is simplified, but latency increases and circuit utilization decreases
Solution Approach 1:
The patent divides the input matrices into smaller blocks (e.g., 4x4 tiles) that can be processed in parallel across multiple circuit blocks. This segmentation allows the system to load and process only the necessary portions of matrices simultaneously, reducing memory access latency while maintaining high circuit utilization through parallel processing of multiple blocks.
Solution Approach 2:
The patent implements a streaming approach where matrix blocks are loaded into the systolic array in advance and processed as they become available. This preliminary action allows the computation to proceed without waiting for complete matrix transfers, reducing latency while keeping circuits busy with continuous processing of incoming blocks.
2Productivity
If hardware accelerators process quantized matrices, then processing efficiency improves, but hardware complexity increases
Solution Approach 1:
The patent applies quantization at the block level rather than requiring full matrix quantization. Each block can be independently quantized and processed with appropriate precision, allowing the hardware to maintain simplicity for unquantized regions while achieving efficiency gains where quantization is beneficial. This localized approach reduces overall hardware complexity.
Solution Approach 2:
The patent supports multiple data precisions (e.g., FP32, FP16, INT8) that can be dynamically selected based on the specific computation requirements. This parameter flexibility allows the hardware to process quantized matrices efficiently when needed while maintaining the ability to handle unquantized data with standard precision, balancing processing efficiency with hardware simplicity.
Data Source
AI summary
A hardware accelerator including tiles arranged in a systolic array. At each of the tiles, the systolic array receives a first input block that includes first input matrix elements of a first input matrix. In each of a plurality of multiplication iterations, at each of the tiles, the systolic array receives a respective second input block. The systolic array computes tile products of the first input matrix elements and second input matrix elements included in the second input blocks. The systolic array adds the tile products to column-wise partial sums and transmits the column-wise partial sums to subsequent tiles along accumulator rings included in array columns of the systolic array. In a subset of the multiplication iterations, the systolic array outputs product block rows of a product matrix. The product block rows each include product matrix blocks computed as rows of the column-wise partial sums.