Sparse DNN Computation Unit for Low-Bandwidth Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing hardware deployment methods for deep neural networks (DNNs) fail to fully utilize structured sparsity, leading to high bandwidth occupation and timing constraints, especially when implementing structured pruning on hardware.
Innovation Solution
A computation unit design that includes a weight buffer, excitation buffer, index selector, and dot product computation unit to efficiently utilize structured sparsity by storing and processing sparsified weight matrices, reducing storage bandwidth and eliminating timing constraints.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a conventional tensor engine or core dedicated for dense DNN inference is used, then the DNN can be implemented on hardware, but structured sparsity of the DNN cannot be fully utilized
Solution Approach 1:
The weight matrix is segmented into non-zero elements only, storing them in a compressed format that exploits the structured sparsity pattern. This segmentation allows the hardware to process only relevant weight elements, achieving both sparsity utilization and computational efficiency
Solution Approach 2:
The storage and computation parameters are changed to accommodate sparse data structures. The system uses specialized data formats and computation pipelines that are parameterized to exploit the known sparsity pattern, enabling efficient hardware implementation that adapts to the structured sparsity of pruned DNNs
2Productivity
If a weight-reuse-based tensor engine design manner is used, then hardware deployment is achieved, but a register file occupies relatively large bandwidth
Solution Approach 1:
Zero-valued weights are extracted and removed from the computation entirely. The system processes only non-zero weight elements, eliminating unnecessary data movement and reducing register file bandwidth requirements while maintaining hardware deployment capability
Solution Approach 2:
The sparsity pattern is analyzed and exploited in advance during the design phase. Weight matrices are pre-processed to identify and remove zero elements before hardware deployment, allowing the register file to be sized appropriately for the actual data volume rather than the full dense matrix
3Productivity
If a weight-reuse-based tensor engine design manner is used, then hardware deployment is achieved, but a timing constraint is relatively strong
Solution Approach 1:
The computation pipeline is designed to be dynamic and adaptable to the sparsity pattern. Rather than rigid timing schedules, the system flexibly processes non-zero elements at varying rates, allowing timing to naturally accommodate the data structure without imposing strict constraints
Data Source
AI summary
This disclosure provides a computation unit, a related apparatus, and a method. The computation unit includes: a weight buffer adapted to store a row vector fetched from an M×Kα sparsified weight matrix, where M and K are respectively a number of rows and a number of columns of the weight matrix before being sparsified, and α is a sparsity coefficient; an excitation buffer adapted to store a K×N excitation matrix; an index selector adapted to store a selection index corresponding to the row vector, and select a row of the excitation matrix based on the selection index, to obtain a Kα×N selected excitation matrix; and a dot product computation unit adapted to multiply the row vector by the selected excitation matrix. This disclosure implements a manner of running a DNN on hardware. In such a manner, structured sparsity of a DNN can be fully utilized, so that inference efficiency is improved; moreover, a register file occupies relatively small bandwidth, and a timing constraint is weak.