Sparse Array Multiplication Accelerator Using Subgroup Masks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current processors struggle to efficiently accelerate array multiplication in machine learning models with structured sparsity, particularly in neural networks, due to fixed compression rates and inefficient utilization of multiply-accumulate units.
Innovation Solution
A hardware accelerator that stores sparse arrays in a compressed format using a mask array and non-zero element array, allowing dynamic selection and multiplication of non-zero elements, optimizing memory usage and performance by reducing unnecessary computations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If sparse arrays are stored in compressed format using mask arrays and non-zero element arrays, then memory footprint is reduced and unnecessary computations are eliminated, but device complexity increases due to additional data structures and loading operations
Solution Approach 1:
The sparse array is segmented into two separate structures: a mask array containing only non-zero elements and their positions, and a corresponding data array. This segmentation eliminates storage of redundant zero elements, reducing memory footprint and enabling the accelerator to process only meaningful data, thereby reducing power consumption while managing complexity through structured organization
Solution Approach 2:
The mask array is pre-computed and stored alongside the sparse array data, identifying all non-zero element positions before the multiplication operation. This preliminary action enables the accelerator to directly load only necessary elements during computation, avoiding runtime analysis of zero elements and reducing both power consumption and operational complexity
2Productivity
If fixed compression rates are used for sparse arrays, then device complexity is reduced with simpler storage schemes, but productivity decreases due to inefficient utilization of multiply-accumulate units
Solution Approach 1:
The compression scheme dynamically adapts to the actual sparsity pattern of each sparse array by computing masks based on real non-zero element distributions rather than applying fixed compression rates. This dynamic approach enables optimal utilization of multiply-accumulate units by ensuring they process only non-zero elements, maximizing productivity while the computed mask structure manages the complexity
3Loss of time
If all elements including zeros are processed in array multiplication, then device complexity is reduced with simpler processing logic, but loss of time increases due to unnecessary multiplication operations
Solution Approach 1:
Zero elements are extracted and excluded from the processing pipeline entirely. The mask array identifies non-zero elements, and the accelerator loads and processes only these elements with the second array. This extraction eliminates wasteful multiplication operations involving zero, reducing computation time while the mask-based selection logic manages processing complexity
Data Source
AI summary
A method and apparatus for multiplying a first array including a plurality of equal-sized subgroups of elements, each including at least a minimum number of zero-elements, by a second array, by, for each subgroup of elements of the first array: loading a subgroup mask indicating locations of non-zero elements within the subgroup of elements of the first array, from memory into a first register; loading, from memory into a second register, the non-zero elements in the subgroup of elements of the first array; loading, from memory into a third register, a subgroup of elements of the second array corresponding to the subgroup of elements of the first array; and multiplying each of the non-zero elements of the first array by the corresponding elements of the second array, wherein the corresponding elements of the second array are selected according to the subgroup mask.