Activation Compression With Block Pruning for Sparse Matrix Multiplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing artificial neural network (ANN) models, particularly deep neural networks (DNNs) and convolutional neural networks (CNNs), face inefficiencies in matrix multiplication due to the presence of sparse matrices, leading to a significant number of 'multiply by zero' operations, which are resource-intensive and inefficient, especially for power or storage-constrained devices.
Innovation Solution
A matrix multiplication system that includes a processor and a matrix multiply accelerator (MMA) to generate basic block matrices, prune elements based on sparsity, apply masks, and compress these matrices, reducing the number of elements to be multiplied, thereby minimizing 'multiply by zero' conditions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If standard matrix multiplication techniques are applied to sparse matrices, then the operation can be performed, but the efficiency is very low due to the large number of operands that have a value of zero
Solution Approach 1:
The patent extracts and removes zero-valued elements from sparse matrices before performing multiplication operations. By identifying and eliminating these redundant zero elements from the input matrices, the system avoids performing unnecessary multiply operations, thereby improving efficiency and reducing resource consumption while maintaining computational accuracy.
Solution Approach 2:
The patent applies partial action by performing multiplication only on the non-zero elements of sparse matrices rather than all elements. This selective approach processes only the necessary portion of the data (non-zero elements), avoiding wasted computational effort on operations that would produce zero results, thus optimizing productivity and reducing energy loss.
2Productivity
If convolution operations are converted into generic matrix multiplication (GEMM) operations, then the calculations can be executed more efficiently by CPU or hardware accelerator, but the data requires expansion into IM2COL format which increases storage requirements
Solution Approach 1:
The patent segments the large expanded IM2COL matrices into smaller basic block matrices (e.g., 4x4 blocks). This segmentation allows the system to process and compress each block independently, identifying and removing zero elements within each block. The segmented approach maintains the computational efficiency of GEMM operations while reducing overall storage requirements through localized compression.
Solution Approach 2:
The patent changes the representation parameters of the matrix data by transitioning from a dense IM2COL format to a compressed format that tracks only non-zero elements. By modifying how the data is stored (using indices and values pairs instead of full matrix expansion), the system achieves both fast GEMM computation and reduced memory footprint.
3Measurement precision
If more elements are retained in the matrix to maintain accuracy, then the computation result is more precise, but the number of operations increases leading to higher resource consumption
Solution Approach 1:
The patent converts the harmful effect of sparse matrices (wasting computational resources on zero multiplications) into a benefit by systematically identifying and exploiting the zero pattern. The sparsity, which initially causes inefficiency, becomes the foundation for optimization algorithms that skip zero operations, thereby reducing processing energy while maintaining the same computational accuracy as full matrix multiplication.
Data Source
AI summary
A system and method for multiplying matrices, and method for training a convolutional neural network (CNN), are provided. The system includes a processor and a matrix multiply accelerator (MMA). The processor is configured to generate, based on an input tensor, a number of basic block matrices, each basic block matrix including a number of elements; for each basic block matrix: prune, based on a sparsity value, the elements of the basic block matrix, generate a mask for the basic block matrix, each mask including a number of bits, each bit corresponding to a different element of the basic block matrix, and compress the basic block matrix to generate a compressed basic block matrix having fewer elements than the basic block matrix. The MMA is configured to multiply, based on the masks, the compressed basic block matrices and a weight matrix to generate an output matrix.