Scalable DNN Accelerator Sparsity Removal for Power Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network processing units (NPUs) for AI/ML applications are memory-size inefficient and power-hungry, lacking flexibility to adapt to evolving machine learning models, and struggle with the high throughput required by real-time applications like autonomous driving, which necessitates efficient sparsity removal in deep neural networks (DNNs).
Innovation Solution
A scalable deep neural networks (DNN) accelerator architecture featuring multiple address generators, an activation memory matrix, and network processing units (NPUs) with activation sparsity removal and redundancy removal blocks, enabling full sparsity removal of zero weights and activations without performance degradation, and allowing independent processing of DNN calculations without synchronization, thus optimizing size and power efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional GPU-based solutions are used for real-time AI applications, then processing throughput can be achieved, but power consumption and memory size increase significantly
Solution Approach 1:
The patent extracts and removes sparse zero-value weights and activations from the neural network computation process. By identifying and eliminating these redundant zero elements before they enter the computation pipeline, the system reduces unnecessary processing operations, thereby lowering power consumption while maintaining the required processing throughput for real-time AI applications.
Solution Approach 2:
The patent changes the computational parameters by dynamically adjusting the precision and representation of neural network weights and activations. By using variable precision arithmetic and adapting the computational format based on the specific layer and operation requirements, the system achieves efficient processing with reduced power consumption compared to traditional fixed-precision GPU approaches.
2Productivity
If traditional GPU-based solutions are used for real-time AI applications, then processing throughput can be achieved, but memory size and cost increase
Solution Approach 1:
The patent extracts sparse zero elements from the neural network data structures and removes them from memory storage and transmission paths. By eliminating these redundant zero values from the computational flow, the system reduces the memory bandwidth requirements and the size of activation maps, thereby reducing overall memory size and associated costs while maintaining processing throughput.
Solution Approach 2:
The patent segments the neural network computation into multiple processing stages, with sparsity removal operations performed at intermediate points between layers. By dividing the computation pipeline and inserting sparsity removal blocks at strategic locations, the system reduces the memory burden at each stage, allowing for more efficient memory utilization and reduced total memory size.
3Productivity
If dedicated ASICs/ASSPs are used for machine learning, then processing efficiency is improved, but flexibility to adapt to evolving models is lost
Solution Approach 1:
The patent implements a universal sparsity removal architecture that can handle multiple neural network architectures, layer types, and sparsity patterns through a single configurable hardware block. The sparsity removal unit is designed to be architecture-agnostic, supporting various deep learning models and evolving computational patterns, thereby providing both high processing efficiency and adaptability to new models without requiring dedicated ASIC designs for each model type.
4Use of energy by moving object
If sparsity removal is implemented in neural network processing, then power consumption and memory size are reduced, but processing complexity increases
Solution Approach 1:
The patent performs sparsity removal as a preliminary operation before the main neural network computation begins. By pre-identifying and removing zero-value weights and activations in advance, the system eliminates redundant computations from the critical path, reducing the overall processing complexity of the neural network while achieving lower power consumption. The sparsity removal block operates independently and prepares optimized data for subsequent processing stages.
Data Source
AI summary
In an example, a scalable deep neural networks (DNN) accelerator (sDNA) includes multiple address generators, an activation memory matrix (AMM), and multiple network processing units (NPUs). The AMM is coupled to outputs of the address generators. The NPUs are coupled to outputs of the AMM. Each NPU includes one of: an activation sparsity removal (ASR) block coupled to the AMM; a redundancy removal (RR) block coupled to the AMM; or both an ASR block coupled to the AMM and an RR block coupled to an output of the ASR block. Each NPU additionally includes a multiply accumulator (MAC) block coupled to the output of the ASR block or an output of the RR block and a non-linear unit coupled to an output of the MAC block.