Weight-Sparse NPU Routing for Mixed Sparsity GEMM
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network accelerators are optimized for specific sparsity categories, leading to inefficiencies when switching between different sparsity models, particularly in terms of area and power consumption, especially when both compute units and RAM are optimized for a specific category.
Innovation Solution
A GEMM accelerator core with a buffer and processing elements that utilize preprocessing to replace zero-valued elements with nonzero-valued elements within a predetermined borrowing distance, generating metadata for efficient operation across all sparsity categories.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If an accelerator is specifically optimized for one sparsity category, then efficiency for that category is improved, but the ability to handle other categories efficiently deteriorates
Solution Approach 1:
The accelerator core is designed with a universal buffer structure that can handle both sparse and dense data formats. The buffer includes routing logic that dynamically selects between sparse and dense access patterns, allowing the same hardware to efficiently process different sparsity categories without requiring separate optimized units for each category.
Solution Approach 2:
The system dynamically adapts its operation mode based on the input data characteristics. The buffer routing mechanism can switch between sparse mode (skipping zero elements) and dense mode (processing all elements) depending on the sparsity pattern detected in the activation or weight tensors, enabling optimal performance across varying workload conditions.
2Area of stationary object
If compute units and RAM are optimized for a specific sparsity category, then area and power consumption are reduced, but overhead for handling other categories increases significantly
Solution Approach 1:
The patent merges sparse and dense processing capabilities into a single unified buffer structure. Instead of having separate optimized units for sparse and dense operations, the design combines both functionalities in one buffer with integrated routing logic, eliminating the need for duplicate hardware resources and reducing overall area while maintaining support for multiple sparsity categories.
Solution Approach 2:
The buffer acts as an intermediary between the compute units and the sparse/dense data formats. It includes routing logic that mediates the data flow, selecting appropriate access patterns based on the input characteristics. This intermediary layer absorbs the complexity of handling multiple formats, allowing the compute units themselves to remain simple and efficient.
3Productivity
If preprocessing is used to replace zero-valued elements with nonzero-valued elements, then efficiency across all sparsity categories is improved, but additional computational overhead is introduced
Solution Approach 1:
The system performs preliminary preprocessing of the weight tensors by replacing zero-valued elements with nonzero-valued elements from the same row or column. This preprocessing generates metadata that enables the buffer routing logic to skip zero elements during inference, improving operational efficiency while the preprocessing overhead is paid only once during model preparation.
Data Source
AI summary
A general matrix-matrix (GEMM) accelerator core includes first and second buffers, and a processing element (PE). The first buffer receives a elements of a matrix A of activation values. The second buffer receives b elements of a matrix B of weight values. The matrix B is preprocessed with a nonzero-valued b element replacing a zero-valued b element in a first row of the second buffer based on the zero-valued b element being in the first row of the second buffer. Metadata is generated that includes movement information of the nonzero-valued b element to replace the zero-valued b element. The PE receives b elements from a first row of the second buffer and a elements from the first buffer from locations in the first buffer that correspond to locations in the second buffer from where the b elements have been received by the PE as indicated by the metadata.