Structured Fine-Grained Sparse Matrix Compression in Systolic Arrays
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AI workloads, particularly those involving large language models and generative models, face challenges in efficiently processing sparse data due to limited techniques for accelerating fine-grained and structured fine-grained sparsity, leading to computational inefficiencies and trade-offs in hardware performance.
Innovation Solution
A hardware accelerator is designed to compress sparse input matrices based on sparsity factors, using multiplexor circuits and index arrays to generate multiplier matrices, while supporting block-scaled data formats, thereby reducing data streaming and enabling efficient matrix multiplication without loss of accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If techniques are used to accelerate fine-grained and structured fine-grained sparse workloads, then processing speed for sparse data is improved, but computational trade-offs occur in accelerating dense workloads on the same hardware
Solution Approach 1:
The hardware accelerator dynamically reconfigures its processing units based on the input data characteristics. When sparse workloads are detected, the system activates specialized sparse processing paths with compression and selective multiplication units. When dense workloads are detected, the system switches to standard matrix multiplication paths, ensuring optimal performance for both workload types without requiring separate dedicated hardware for each.
Solution Approach 2:
The accelerator employs universal processing elements that can handle both sparse and dense matrix operations. The same hardware units perform different functions depending on the operation mode: for sparse operations, they execute compressed data processing with skip logic for zero elements; for dense operations, they perform full matrix multiplication. This multi-functionality eliminates the need for separate hardware accelerators for sparse and dense workloads.
2Quantity of substance
If sparse data is compressed to remove zero-valued elements, then data volume is reduced and processing efficiency is improved, but additional computational overhead is introduced for compression and decompression
Solution Approach 1:
The system performs compression of sparse matrices before they enter the processing pipeline. By pre-compressing the input data and storing it in a compact format with associated metadata (row pointers, column pointers, value arrays), the system reduces the amount of data that needs to be processed throughout the computation. The decompression occurs incrementally during the multiplication process rather than all at once, spreading the computational overhead across multiple operation cycles.
Solution Approach 2:
The compression mechanism extracts only the non-zero elements from sparse matrices, storing them in a compact format while maintaining the ability to reconstruct the original matrix structure. This extraction of essential information (non-zero values and their positions) eliminates redundant zero element storage and processing, reducing data volume while preserving computational accuracy.
3Quantity of substance
If block quantization is applied to represent data with shared scaling factors, then memory usage is reduced and processing throughput is improved, but precision is lost due to approximation
Solution Approach 1:
The system changes the precision parameter of data representation by applying block quantization, where groups of floating-point values are converted to lower-precision formats (e.g., INT8, INT4) with shared scaling factors. This parameter change reduces memory bandwidth requirements and increases processing throughput. The scaling factors are carefully selected to minimize precision loss, and the system supports multiple precision levels to balance between accuracy and performance based on specific workload requirements.
4Productivity
If SIMD processing is used to perform operations on multiple data inputs in parallel, then processing throughput is improved, but data must be transferred through processing lanes which increases complexity
Solution Approach 1:
The SIMD processor divides the data processing task into multiple parallel processing lanes, each handling a subset of the input data simultaneously. For sparse matrix operations, each lane processes corresponding elements from different rows or columns in parallel. This segmentation enables throughput improvement by keeping all processing units actively engaged, while the regular lane structure simplifies data transfer compared to more complex interconnection networks.
Data Source
AI summary
Methods, systems, and apparatus, including computer-readable storage media for hardware-accelerated fine-grained sparse computation. The accelerator provides for improved performance for structured fine-grained sparse AI workloads, for example by accelerating sparse matrix multiplication required to execute or train AI models. Sparse data is compressed to remove zero-valued elements before being streamed into a matrix multiplication unit (MXU) of the accelerator. The accelerator stores a gains matrix, which can be the matrix for multiplying with the received input matrix. The accelerator uses an index array mapping locations of elements in the compressed matrix with locations of elements in the matrix's pre-compressed form, to generate a multiplier matrix from the gains matrix. Aspects of the disclosure also provide for accelerated gains matrix loading in a hardware accelerator or other type of processor. The accelerator can load the gains matrix more efficiently in a compressed form, and then un-compress the matrix once loaded.