Sparse-Dense Matrix Accelerator Using Compressed Nonzero Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional CPU and GPU instruction set architectures require symmetric inputs for sparse-dense matrix multiplication, limiting performance gains from sparse input matrix sparsity.
Innovation Solution
An accelerator for sparse-dense matrix multiplication (SDMM) that processes asymmetric inputs using SIMD processing lanes, multi-bank memories, and compressed sparse row/column formats to improve throughput by up to 20-fold.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional CPU and GPU instruction set architectures are used for sparse-dense matrix multiplication, then symmetric inputs are required, but performance gains from sparse input matrix sparsity are limited
Solution Approach 1:
The patent applies asymmetry by designing an SDMM accelerator that accepts asymmetric inputs where the sparse matrix A and dense matrix B can have different dimensions and sparsity patterns. The architecture uses separate storage structures for sparse and dense matrices, allowing independent optimization of each input type without requiring symmetric properties, thereby enabling performance gains proportional to the sparsity of matrix A
Solution Approach 2:
The patent segments the matrix multiplication operation into distinct handling paths for sparse and dense matrices. The SDMM accelerator divides the computation into: (1) identifying non-zero elements in sparse matrix A, (2) retrieving corresponding rows from dense matrix B, (3) performing multiplication only for non-zero elements, and (4) accumulating results. This segmentation eliminates wasted computation on zero elements and achieves throughput improvements of up to 20-fold for highly sparse matrices
2Productivity
If traditional symmetric matrix multiplication is used, then processing cycles are wasted on zero-valued elements, but the architecture remains simple
Solution Approach 1:
The patent extracts only the non-zero elements from the sparse matrix for processing, storing them in a compressed format that separates non-zero data from zero elements. The SDMM accelerator retrieves and processes only these extracted non-zero elements along with their corresponding indices, completely eliminating processing cycles wasted on zero-valued elements while maintaining a manageable architecture through dedicated sparse matrix storage and access mechanisms
Solution Approach 2:
The patent applies preliminary action by pre-processing the sparse matrix into a compressed format that identifies and stores only non-zero elements and their positions before the multiplication operation. This pre-processing creates index structures and compressed representations that enable the SDMM accelerator to directly access and process only relevant non-zero elements during computation, avoiding the need to handle zero elements during the actual multiplication phase
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
Disclosed embodiments relate to multiply-accumulate operations. In one example, a processor, comprises fetch circuitry, a plurality of registers, and execution circuitry. The fetch circuitry is to fetch a sparse-dense matrix multiplication (SDMM) instruction from a memory. The SDMM instruction has fields to specify an opcode to indicate a sparse-dense matrix multiplication operation, a result matrix having matrix data element dimensions of M x N, a first source matrix having source matrix data element dimensions of K x N, and a second source matrix representing a sparse source matrix having source matrix data element dimensions of M x K. Matrix data elements in the K dimension of the sparse source matrix comprise zero-value data elements and remaining data elements. In the K dimension, the second source matrix is to include the remaining data elements having position values associated therewith. The plurality of registers is to store a plurality of source data elements of the first source matrix, to store the remaining data elements of the second source matrix, and to store the position values associated with the remaining data elements, each position value associated with a location of a corresponding one of the remaining data elements within the sparse source matrix. The execution circuitry, responsive to the SDMM instruction, is to multiply the remaining data elements with corresponding data elements of the first source matrix in accordance with the position values to produce a plurality of products, and to accumulate subsets of the plurality of products to produce corresponding data elements of the result matrix.