Block-wise N:M Sparsity for DNN Accelerator Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep neural network (DNN) acceleration techniques face inefficiencies in computation and memory usage due to fine-grained sparsity, which is not hardware-friendly and incurs high overhead costs.
Innovation Solution
A software/hardware co-design technique that induces block-wise sparsity in DNN weights, utilizing a coarse-grain 2:4 sparsity pattern to reduce computational complexity and memory footprint, allowing for efficient hardware acceleration with low overhead circuitry.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If fine-grained sparsity is used to achieve high sparsity ratio, then computation complexity is reduced, but hardware friendliness deteriorates due to high overhead costs
Solution Approach 1:
The weight matrix is segmented into blocks of size (k,c) where k>=2, organizing sparse weights into structured blocks. This segmentation allows the hardware to process blocks collectively rather than individual elements, reducing control overhead while maintaining high sparsity ratios through block-wise skip logic.
Solution Approach 2:
The sparsity pattern changes from fine-grained (element-level) to coarse-grained (block-level) by modifying the organization parameter from individual weight elements to kx c blocks. This parameter change enables hardware-friendly processing where entire blocks can be skipped or processed together, reducing per-element overhead.
2Ease of operation
If block-wise sparsity with block size (k,c) is used where k>=2, then hardware friendliness is improved, but memory organization complexity increases
Solution Approach 1:
Weights are segmented into regular kx c blocks that align with processing element groups. This regular segmentation creates predictable memory access patterns where each PE group accesses a known block structure, simplifying memory organization despite the block-wise sparsity pattern.
Solution Approach 2:
The block structure serves multiple functions: it defines sparsity patterns, organizes memory layout, and groups processing elements. This universal block concept simplifies overall system design by using the same structural unit across different subsystems, offsetting the apparent memory organization complexity.
3Use of energy by moving object
If N:M block-wise structured sparsity is applied, then area/power efficiency is improved by skipping ineffectual multiplications, but computation complexity increases
Solution Approach 1:
Zero-weight blocks are extracted and removed from the computation path entirely. The N:M sparsity pattern identifies M blocks containing zeros and excludes them from processing, so processing elements only compute for N non-zero blocks. This extraction eliminates wasteful multiplications and reduces active computation logic.
Solution Approach 2:
The hardware skips over blocks known to contain zero weights using the N:M sparsity pattern. Control logic rapidly identifies and skips M zero blocks to reach N active blocks, rushing through ineffective computation regions and focusing resources only on productive multiplications, thereby improving energy efficiency.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
An accelerator core includes first and second buffers and at least one group of k processing elements. The first buffer receives at least one group of block-wise sparsified first elements. A block size (k,c) of each group of block-wise sparsified first elements includes k rows and c columns in which k is greater than or equal to 2, k times p equals K, and c times q equals C in which K is an output channel dimension of a tensor of first elements, C is a number of input channels of the tensor of first elements, p is an integer and q is an integer. The second buffer receive second elements. Each respective group of processing elements receive k rows of first elements from a block of first elements corresponding to the group of PEs, and receives second elements that correspond to first elements received from the first buffer.