Block-Scaled Sparse Matrix Processing in Systolic Arrays
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AI workloads, particularly those involving large language models and generative models, face challenges in efficiently processing sparse data due to limited techniques for accelerating fine-grained and structured fine-grained sparsity, leading to computational inefficiencies and trade-offs in hardware performance.
Innovation Solution
A hardware accelerator is designed to compress sparse input matrices based on sparsity factors, using multiplexor circuits and index arrays to generate multiplier matrices, while supporting block-scaled data formats, thereby reducing data streaming and enabling efficient matrix multiplication without loss of accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If techniques are used to accelerate fine-grained and structured fine-grained sparse workloads, then processing speed for sparse data is improved, but computational trade-offs occur in accelerating dense workloads on the same hardware
Solution Approach 1:
The hardware accelerator dynamically reconfigures its processing units based on the input data characteristics. When sparse workloads are detected, the system activates specialized sparse processing paths with compression and skip logic. When dense workloads are detected, the system switches to standard processing modes, eliminating the need for separate dedicated hardware for each workload type.
Solution Approach 2:
The patent implements a universal processing architecture that can handle both sparse and dense workloads through the same hardware infrastructure. The processing units are designed with configurable pathways that adapt to the sparsity factor of input data, allowing a single hardware system to serve multiple function types without requiring separate dedicated accelerators.
2Productivity
If sparse data is processed using traditional methods, then hardware simplicity is maintained, but computational inefficiency occurs due to processing zero-valued elements
Solution Approach 1:
The hardware accelerator extracts and processes only the non-zero elements of sparse matrices by detecting sparsity patterns and skipping zero-valued elements entirely. The system identifies the positions and values of non-zero elements and routes them through specialized processing pathways, eliminating wasted computational cycles on zero multiplications while adding minimal detection logic.
Solution Approach 2:
The patent applies different processing strategies to different regions of the data based on local sparsity characteristics. Within matrices, the system identifies regions with varying sparsity factors and applies optimized processing techniques locally - using compression for highly sparse regions and standard processing for less sparse regions, thereby improving overall efficiency without requiring complete architectural overhaul.
3Quantity of substance
If block quantization is applied to reduce data precision, then memory usage and processing bandwidth are reduced, but data accuracy is lost
Solution Approach 1:
The hardware accelerator dynamically adjusts the precision and block size parameters based on the sparsity factor and computational requirements of the workload. For highly sparse data, the system uses lower precision representations with smaller block sizes to maximize compression. For less sparse or accuracy-critical computations, the system increases precision and block sizes, allowing flexible optimization of the trade-off between data volume and accuracy.
Data Source
AI summary
Methods, systems, and apparatus, including computer-readable storage media for processing block-scaled data on processing devices, where the block size of the data is smaller than the number of implemented processing lanes on the devices. An example process performed by the devices is matrix multiplication. The processing device is configured to load pre-computed scaling factors for static data, and to generate scaling factors for dynamic data as part of the matrix multiplication pipeline for the device. The processing device is configured to cause scaling factors of different blocks of either operand matrix being multiplied to be applied to corresponding blocks during multiplication. The processing device can generate correct matrix multiplication products of block-scaled input, even when the block size is more granular or smaller than the number of processing lanes. Aspects of the disclosure relate to generating scaling factors for input matrices received by a SIMD-configured processing device.