Sparse Neural Network Hardware Architecture for Inactive-Value Skipping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The exponential growth in the complexity of artificial neural networks (ANNs) outpaces hardware performance, leading to bottlenecks in training and inference due to massive linear tensor operations, particularly multiplication and convolution, which consume significant CPU bandwidth and power.
Innovation Solution
An AI accelerator with a memory circuit and sparsity processing circuit that identifies active values in sparse weight tensors, performing efficient computations by skipping inactive values and utilizing a block or partitioned structure to accelerate operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a CPU is used to execute neural network operations, then the system is flexible and easy to update, but the CPU bandwidth consumption and power consumption increase significantly
Solution Approach 1:
The system is segmented into two parts: a CPU that handles high-level control and model updates, and dedicated AI accelerator hardware that handles computationally intensive neural network operations. This segmentation allows the CPU to remain flexible while the accelerator handles the energy-intensive computations efficiently.
Solution Approach 2:
An AI accelerator acts as an intermediary between the CPU and the neural network computations. The accelerator receives model parameters and data from the CPU, performs the computationally intensive operations, and returns results to the CPU, thereby reducing CPU bandwidth consumption and power usage.
2Reliability
If dense neural network operations are performed, then comprehensive computations are executed, but the number of operations and computational resources required increase
Solution Approach 1:
The system extracts and processes only the relevant active values from sparse tensors, removing unnecessary zero-value computations. The sparsity processing circuit identifies and extracts only the non-zero elements that contribute to the final result, thereby maintaining computation completeness while improving efficiency.
Solution Approach 2:
The system changes the parameter representation from dense tensors to sparse tensors, where only active (non-zero) values are stored and processed. This parameter change reduces the number of operations required while maintaining the reliability of computations that depend on active values.
3Productivity
If sparse tensor processing is implemented, then the number of operations is reduced, but additional processing circuits and complexity are required
Solution Approach 1:
The sparsity processing circuit performs preliminary actions by pre-identifying and locating active values in sparse tensors before the main computation occurs. This preliminary processing of sparsity patterns allows subsequent computations to efficiently skip inactive values without requiring complex runtime decisions.
Solution Approach 2:
The system creates specialized copies of processing circuits (multiply circuits, sparsity processing circuits) that are optimized for sparse tensor operations. These copied circuits are replicated and configured to handle different aspects of sparse computation, providing efficiency gains while managing complexity through modular replication.
Data Source
AI summary
A hardware accelerator that is efficient at performing computations related to a sparse neural network. The sparse neural network may be associated with a plurality of nodes. One of the nodes includes one or more sparse tensors. The accelerator may compress the sparse tensor to a dense tensor. The sparse tensor may also be structured so that the dense locations in the tensor are blocked or partitioned. The accelerator may transpose the weight tensor and align the partitions of the tensor with the hardware architecture. The structured tensor has a balanced number of active values so that the active values can be processed by an efficient number of operating cycles of the accelerator. The accelerator may also perform bitwise and operation to determine the location of dense pairs in two sparse tensors to reduce the number of computations.