Sparse Edge Inference Accelerator for Variable Tensor Sparsity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks (DNNs) require efficient acceleration techniques, particularly in exploiting sparsity within operands of General Matrix Multiply (GEMM) operations to maintain accuracy and reduce hardware overhead, where structured sparsity offers advantages over random sparsity but existing solutions do not effectively handle varying sparsity densities.
Innovation Solution
A neural network inference accelerator architecture that includes a memory, neural processing units, and a sparsity management unit, capable of handling both structured and random sparsity densities, with compressor and decompressor units to manage and process tensors based on predetermined sparsity thresholds, allowing for efficient computation and storage of activation and weight tensors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If structured sparsity is used to accelerate NPU operations, then computational efficiency is improved, but hardware overhead increases
Solution Approach 1:
The system dynamically reconfigures NPU hardware architecture based on the sparsity density of input tensors. When high sparsity is detected, the system activates specialized sparse computation units with reduced hardware overhead. When sparsity is low, it switches to dense computation modes, optimizing the balance between computational efficiency and hardware resource usage.
Solution Approach 2:
The invention changes the sparsity density parameter threshold dynamically. By adjusting the threshold for what constitutes 'sparse' versus 'dense' computation, the system can adapt its hardware configuration to match the actual data characteristics, thereby improving computational efficiency without incurring unnecessary hardware overhead for all operations.
2Productivity
If sparsity exploitation is applied to all tensors, then acceleration is improved, but accuracy deteriorates for tensors with moderate sparsity
Solution Approach 1:
The system applies different computation strategies to different tensors based on their individual sparsity characteristics. Rather than uniformly applying sparsity exploitation to all tensors, the system locally adapts its approach - using sparse computation only for tensors that meet the sparsity threshold, and dense computation for others, thereby preserving accuracy where needed while gaining acceleration where applicable.
Solution Approach 2:
The sparsity threshold itself is made dynamic rather than fixed. The system continuously evaluates tensor sparsity patterns and adjusts the threshold adaptively, allowing it to capture acceleration opportunities when they exist while maintaining accuracy when sparsity is insufficient to justify the computational trade-off.
3Productivity
If high sparsity density threshold is set, then more tensors qualify for sparse processing, but hardware complexity increases
Solution Approach 1:
The NPU is designed with multi-functional computation units that can handle both sparse and dense operations. Rather than requiring separate dedicated hardware for sparse processing, the same hardware resources are configured to perform sparse computations when needed and dense computations otherwise, reducing overall hardware complexity while maintaining broad sparse processing coverage.
4Device complexity
If low sparsity density threshold is set, then hardware overhead is reduced, but fewer tensors benefit from acceleration
Solution Approach 1:
The system dynamically adjusts the sparsity threshold based on runtime conditions and tensor characteristics. When hardware resources are available and tensors exhibit moderate sparsity, the threshold is lowered to capture more acceleration opportunities. When resources are constrained, the threshold rises to reduce overhead, creating a dynamic balance between acceleration coverage and hardware utilization.
Data Source
AI summary
A neural network inference accelerator includes first and second neural processing units (NPUs) and a sparsity management unit. The first NPU receives activation and weight tensors based on an activation sparsity density and a weight sparsity density both being greater than a predetermined sparsity density. The second NPU receives activation and weight tensors based on at least one of the activation sparsity density and the weight sparsity density being less than or equal to the predetermined sparsity density. The sparsity management unit controls transfer of the activation tensor and the weight tensor based on the activation sparsity density and the weight sparsity density with respect to the predetermined sparsity density.