Block-Scaled Structured Sparse Matrix Acceleration in Systolic Arrays
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AI workloads, particularly those involving large language models and generative models, face challenges in efficiently processing sparse data due to limited techniques for accelerating fine-grained and structured fine-grained sparsity, leading to computational inefficiencies and trade-offs in hardware utilization.
Innovation Solution
A hardware accelerator is designed to compress sparse input matrices based on sparsity factors, using multiplexor circuits and index arrays to generate multiplier matrices, while supporting block-scaled data formats, thereby reducing data streaming and enabling efficient matrix multiplication without loss of accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If techniques are used to accelerate fine-grained and structured fine-grained sparse workloads, then processing speed for sparse data is improved, but hardware utilization and computational efficiency deteriorate due to limited techniques and specific sparsity factor requirements
Solution Approach 1:
The hardware accelerator divides the processing array into multiple segments, each capable of handling different sparsity patterns. The segmentation allows the system to process various types of sparse workloads (coarse-grained, fine-grained, structured fine-grained) simultaneously without requiring dedicated hardware for each type, thereby improving both speed and hardware utilization.
Solution Approach 2:
The accelerator implements a universal processing architecture that can handle multiple workload types (dense and various sparse patterns) using the same hardware resources. By making the hardware multi-functional, the system achieves high processing speed for sparse workloads while maintaining high hardware utilization through versatile resource usage.
2Loss of time
If data is streamed in compressed form to reduce data loading, then data loading time is reduced, but processing accuracy may deteriorate due to compression losses
Solution Approach 1:
The system performs preliminary decompression of compressed data before it enters the processing array. This preliminary action ensures that the full-precision data is available for accurate computation while still benefiting from the reduced data loading time achieved through compression. The decompression is orchestrated to occur in advance of the computational phase.
Solution Approach 2:
The patent introduces an intermediary decompression stage between the compressed data input and the processing array. This intermediary component acts as a mediator that converts compressed data into full-precision format, allowing the system to享受 both the speed benefits of compression and the accuracy benefits of full-precision processing.
3Adaptability or versatility
If the same hardware is used for both dense and sparse workload multiplication, then hardware versatility is improved, but processing efficiency deteriorates due to computational trade-offs
Solution Approach 1:
The hardware accelerator implements dynamic configuration capabilities that allow processing elements to adapt their operation mode based on the input workload characteristics. The system can dynamically switch between dense and sparse processing modes, optimizing efficiency for each workload type while maintaining versatility. This dynamic adaptation resolves the trade-off by making the hardware efficient for both workload types sequentially.
Data Source
AI summary
Aspects of the disclosure are directed to hardware acceleration of structured sparse workloads with block quantization. A hardware accelerator can receive compressed input matrices, for example as part of a workload for training or processing a machine learning model. The hardware accelerator can multiply the compressed input matrix with a gains matrix loaded in one or more matrix multiply units (MXUs) of the hardware accelerator. The input matrices can be further provided in a block data type format, in which blocks of mantissas are represented with a single shared scaling factor. An MXU can multiply the block data, shift or cast the block data according to a shared scaling factor to generate an output product. To that end, block data type matrices exhibiting structured sparsity patterns can be accelerated without affecting the overall accuracy or quality of the output to the workload being processed.