AI Accelerator Activation Sparsity for Sparse Neural Network Throughput
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The exponential growth in the complexity of artificial neural networks (ANNs) outpaces hardware performance, leading to bottlenecks in training and inference due to massive linear tensor operations, which consume significant CPU bandwidth and power.
Innovation Solution
An AI accelerator with a memory circuit, multiply circuit, and comparator circuit is designed to process sparse neural networks by reducing the number of active values in output activation tensors using methods like K-winner approaches, sorting, and thresholding to enhance efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If neural network complexity increases exponentially, then model accuracy and capability improve, but hardware performance bottlenecks worsen and computational resources are overwhelmed
Solution Approach 1:
The patent extracts and removes inactive values (zeros) from the output activation tensor, keeping only the top K active values. This extraction reduces the data volume that needs to be processed in subsequent layers, directly addressing the computational throughput bottleneck while preserving the essential information needed for model accuracy.
Solution Approach 2:
The patent applies different sparsity levels to different parts of the neural network by allowing K to vary across nodes and layers. This local quality approach optimizes computational resources by applying higher sparsity where appropriate while maintaining necessary detail in critical regions, balancing model accuracy with computational efficiency.
2Ease of operation
If CPU is used to execute neural network operations, then flexibility and ease of implementation are maintained, but CPU bandwidth consumption increases and power consumption rises
Solution Approach 1:
The patent changes the data representation parameter by transforming dense activation tensors into sparse tensors with only K active values. This parameter change reduces the bandwidth requirements and power consumption of CPU operations while maintaining implementation flexibility through software-based sparsity management.
Solution Approach 2:
The patent segments the activation tensor into active and inactive portions, processing only the active values through the network. This segmentation reduces the computational load on the CPU, lowering power consumption and bandwidth usage while maintaining the ability to implement various neural network architectures.
3Loss of information
If all values in output activation tensor are processed, then complete information is preserved, but computational time and resources are wasted on inactive values
Solution Approach 1:
The patent extracts only the top K active values from the output activation tensor based on their magnitude. This extraction preserves the most significant information while eliminating redundant inactive values, thereby reducing computational time without causing significant information loss.
Solution Approach 2:
The patent applies partial action by processing only a subset (K values) of the complete activation tensor rather than all values. This partial processing reduces computational time while maintaining sufficient information for accurate network operation, accepting that not all original values are equally important.
Data Source
AI summary
A hardware accelerator that is efficient at performing computations related to a sparse neural network. The sparse neural network may be associated with a plurality of nodes. An artificial intelligence (AI) accelerator stores, at a memory circuit, a weight tenor and an input activation tensor that corresponds to a node of the neural network. The AI accelerator performs a computation such as convolution between the weight tenor and the input activation tensor to generate an output activation tensor. The AI accelerator introduces sparsity to the output activation tensor by reducing the number of active values in the output activation tensor. The sparsity activation may be a K-winner approach, which selects the K-largest values in the output activation tensor and set the remaining values to zero.