Neural Network Processing Units Activation Sparsity Removal
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current AI/ML processing units, such as GPUs and ASICs, are memory-size inefficient and lack flexibility to adapt to evolving machine learning models, leading to high power consumption and cost due to inefficient handling of sparsity in neural network operations.
Innovation Solution
The implementation of a scalable DNN accelerator (sDNA) architecture with non-zero Activation jump algorithm and multiple FIFO memories to store non-zero activations for efficient vector multiplication, enabling full removal of zero weights and activations without performance degradation, and allowing independent processing unit operations without synchronization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional GPU/CPU/ASIC architectures are used for AI/ML processing, then processing throughput can be achieved, but memory size efficiency deteriorates and power consumption increases due to inability to handle sparsity effectively
Solution Approach 1:
The patent extracts and removes zero-valued activations from the computation pipeline entirely. By implementing sparsity detection and removal mechanisms, the system eliminates unnecessary memory accesses and computations involving zero values, thereby reducing power consumption while maintaining processing throughput for meaningful data.
Solution Approach 2:
The system dynamically adapts its processing behavior based on the sparsity characteristics of the input data. It dynamically adjusts memory access patterns, computation scheduling, and resource allocation to match the actual data distribution, optimizing power efficiency without sacrificing throughput.
2Productivity
If traditional GPU/CPU/ASIC architectures are used for AI/ML processing, then processing throughput can be achieved, but memory size efficiency deteriorates due to inefficient handling of sparsity
Solution Approach 1:
The patent extracts and removes zero-valued activations from the computation pipeline entirely. By implementing sparsity detection and removal mechanisms, the system eliminates unnecessary memory accesses and computations involving zero values, thereby reducing power consumption while maintaining processing throughput for meaningful data.
Solution Approach 2:
The system changes the representation parameters of activation data by transforming dense activations into sparse representations. This parameter change enables more efficient memory utilization by storing and processing only non-zero values, improving memory size efficiency while maintaining processing capability.
3Productivity
If dedicated ASICs/ASSPs are used for AI/ML processing, then processing throughput can be achieved, but adaptability deteriorates due to inability to adapt to changing machine learning models
Solution Approach 1:
The patent implements a universal processing architecture that can handle multiple machine learning models and workloads. By incorporating programmable components and standardized interfaces, the system achieves high throughput for various models while maintaining the flexibility to adapt to changing requirements, unlike dedicated ASICs.
Solution Approach 2:
The system dynamically adapts its processing behavior based on the sparsity characteristics of the input data. It dynamically adjusts memory access patterns, computation scheduling, and resource allocation to match the actual data distribution, optimizing power efficiency without sacrificing throughput.
4Use of energy by stationary object
If sparsity removal is implemented in neural network processing, then size and power efficiency improve, but processing complexity increases
Solution Approach 1:
The patent performs sparsity detection and removal as a preliminary action before the main computation pipeline. By identifying and eliminating zero-valued activations in advance, the system reduces the complexity of subsequent processing stages while achieving power efficiency benefits from reduced computation and memory access.
Data Source
AI summary
In an example, a method of activation sparsity removal includes implementing a non-zero Activation jump algorithm. Alternatively, the method includes using multiple first in first out (FIFO) memories to store non-zero activations for each vector multiplication.