Structured Sparse Matrix Compression for Systolic Array Acceleration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AI workloads, particularly those involving large language models, face challenges in efficiently processing sparse data due to limited techniques for accelerating fine-grained and structured fine-grained sparsity, leading to computational inefficiencies and trade-offs in hardware resources.
Innovation Solution
A hardware accelerator is designed to compress sparse input matrices based on sparsity factors, using multiplexor circuits and index arrays to generate multiplier matrices, while supporting block-scaled data formats, thereby reducing data streaming and enabling efficient matrix multiplication without loss of accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If techniques are used to accelerate fine-grained and structured fine-grained sparse workloads, then processing speed for sparse data is improved, but hardware resources and computational efficiency deteriorate due to limited acceleration techniques and specific sparsity factor requirements
Solution Approach 1:
The hardware accelerator is designed with a universal architecture that can handle multiple types of workloads (dense, coarse-grained sparse, fine-grained sparse, and structured fine-grained sparse) using the same data path and computational units. The system uses configurable sparsity handling mechanisms that can be adjusted based on the input data characteristics, eliminating the need for separate dedicated hardware for each workload type.
Solution Approach 2:
The accelerator employs dynamic sparsity detection and handling mechanisms that adapt to the actual sparsity patterns in the input data. The system can dynamically switch between different processing modes (dense mode, sparse mode with various sparsity factors) based on the detected characteristics of the incoming workload, optimizing resource utilization for each specific case.
2Quantity of substance
If sparse data compression is applied, then data streaming requirements are reduced, but processing complexity increases due to the need for multiplexor circuits and index arrays
Solution Approach 1:
The system performs preliminary compression of sparse data before it enters the main computation pipeline. By pre-compressing the input matrices and pre-computing index arrays that map compressed elements to their original positions, the system reduces the data volume that needs to be streamed and processed, while the compression overhead is paid once before the main acceleration benefit is realized.
Solution Approach 2:
Index arrays serve as intermediary structures that bridge the compressed data format and the original data layout required for computation. These index arrays enable the hardware to efficiently access and process only the non-zero elements of sparse matrices without requiring complex decompression circuits, by directly mapping compressed element positions to their computational roles.
3Quantity of substance
If block quantization is used to represent data, then memory efficiency is improved, but precision is reduced due to lossy approximation
Solution Approach 1:
The system changes the precision parameter of data representation by applying block quantization, where blocks of floating-point values are approximated using lower-precision formats (such as INT8 or INT4) with shared scaling factors. This parameter change reduces memory storage requirements and increases data throughput, while the lossy approximation is managed through careful scaling factor selection and post-processing to minimize accuracy degradation.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
Methods, systems, and apparatus, including computer-readable storage media for hardware-accelerated fine-grained sparse computation. The accelerator provides for improved performance for structured fine-grained sparse AI workloads, for example by accelerating sparse matrix multiplication required to execute or train AI models. Sparse data is compressed to remove zero-valued elements before being streamed into a matrix multiplication unit (MXU) of the accelerator. The accelerator stores a gains matrix, which can be the matrix for multiplying with the received input matrix. The accelerator uses an index array mapping locations of elements in the compressed matrix with locations of elements in the matrix's precompressed form, to generate a multiplier matrix from the gains matrix. Aspects of the disclosure also provide for accelerated gains matrix loading in a hardware accelerator or other type of processor. The accelerator can load the gains matrix more efficiently in a compressed form, and then un-compress the matrix once loaded.