Systolic Array Sparse Dot Product Using Structured Sparsity Metadata
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systolic arrays struggle to effectively utilize structured sparsity in neural network workloads, leading to inefficient data movement and reduced compute density due to the lack of hardware resources and design complexity in handling preprocessed sparse data.
Innovation Solution
Implement a new instruction (sparse systolic dot product with accumulate, or SDPAS) that recognizes and leverages structured sparsity in input data, utilizing an array of selection circuits and metadata to optimize matrix multiplication operations in systolic arrays.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional systolic arrays are used without structured sparsity support, then hardware resources and design complexity remain simple, but data movement efficiency and compute density deteriorate
Solution Approach 1:
The systolic array is segmented into multiple stages, with each stage equipped with selection circuits that can independently select between sparse and dense data paths. This segmentation allows the system to process sparse data efficiently without requiring complete redesign of the entire array, thus improving data movement efficiency while controlling hardware complexity growth.
Solution Approach 2:
The patent introduces a new dimension of operation by adding selection circuits that operate in parallel with the existing compute units. This dimensional addition enables the system to handle structured sparsity without fundamentally altering the core systolic array architecture, thereby improving productivity with moderate increases in device complexity.
2Productivity
If structured sparsity is supported with additional hardware, then compute density improves, but device complexity increases
Solution Approach 1:
The selection circuits are designed to be multi-functional, serving both as data selectors for sparse operations and as regular data pass-through elements for dense operations. This universality allows the hardware to achieve higher compute density when processing sparse data while maintaining compatibility with conventional workloads, thus improving productivity without proportionally increasing device complexity.
Solution Approach 2:
The systolic array stages are equipped with self-contained selection circuits that autonomously determine whether to perform sparse or dense operations based on the input data characteristics. This self-service capability eliminates the need for complex external control logic, thereby improving compute density while minimizing the increase in overall device complexity.
3Loss of time
If sparse data is processed without optimization, then data movement remains high, but execution time increases
Solution Approach 1:
The selection circuits extract and isolate the non-zero elements from sparse data, routing them through optimized computation paths while eliminating unnecessary data movement of zero elements. This extraction mechanism directly reduces the quantity of data that needs to be moved and processed, thereby decreasing execution time without requiring excessive hardware resources.
Solution Approach 2:
The systolic array stages perform preliminary selection of relevant data elements before they enter the computation units. By pre-filtering out zero elements and routing only necessary data through the computation pipeline, the system minimizes data movement and reduces execution time, achieving efficient sparse matrix multiplication with controlled hardware complexity.
Data Source
AI summary
An apparatus to facilitate utilizing structured sparsity in systolic arrays is disclosed. The apparatus includes a processor comprising a systolic array to receive data from a plurality of source registers, the data comprising unpacked source data, structured source data that is packed based on sparsity, and metadata corresponding to the structured source data; identify portions of the unpacked source data to multiply with the structured source data, the portions of the unpacked source data identified based on the metadata; and output, to a destination register, a result of multiplication of the portions of the unpacked source data and the structured source data.