Sparse Neural Network Accelerator Architecture for Active-Value Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The exponential growth in the complexity of artificial neural networks (ANNs) outpaces hardware performance, leading to bottlenecks in training and inference due to massive linear tensor operations, particularly multiplication and convolution, which consume significant CPU bandwidth and increase power consumption.
Innovation Solution
An AI accelerator is designed to efficiently process sparse neural networks by utilizing a memory circuit, sparsity processing circuit, and multiply circuit to identify and compute only active values in sparse weight tensors, with a structure that limits active value distribution patterns, and applies a K-winner activation function to generate a sparse output.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If CPU and main memory are used to execute machine learning systems, then ease of instantiation is improved, but CPU bandwidth consumption and power consumption increase significantly
Solution Approach 1:
The system divides the machine learning computation into separate functional segments: a CPU that handles high-level model instantiation and control, and a dedicated AI accelerator that handles computational operations. This segmentation allows the CPU to remain lightweight while the accelerator absorbs the computational workload, resolving the contradiction between ease of instantiation and power consumption.
Solution Approach 2:
A dedicated AI accelerator circuit is introduced as an intermediary between the CPU and the computational workload. This accelerator includes specialized components (sparsity processing circuit, multiply circuits, memory circuits) that mediate the execution of machine learning operations, offloading the CPU from bandwidth-intensive tasks while maintaining system functionality.
2Ease of operation
If CPU and main memory are used to execute machine learning systems, then ease of instantiation is improved, but CPU bandwidth consumption increases significantly
Solution Approach 1:
The system divides the machine learning computation into separate functional segments: a CPU that handles high-level model instantiation and control, and a dedicated AI accelerator that handles computational operations. This segmentation allows the CPU to remain lightweight while the accelerator absorbs the computational workload, resolving the contradiction between ease of instantiation and power consumption.
Solution Approach 2:
A dedicated AI accelerator circuit is introduced as an intermediary between the CPU and the computational workload. This accelerator includes specialized components (sparsity processing circuit, multiply circuits, memory circuits) that mediate the execution of machine learning operations, offloading the CPU from bandwidth-intensive tasks while maintaining system functionality.
3Reliability
If dense neural networks are used, then inference accuracy is maintained, but training and inference time increase due to massive linear tensor operations
Solution Approach 1:
The system extracts and processes only the essential computational operations from dense network executions. By identifying and isolating critical paths and active values in tensor operations, the accelerator can process only the necessary computations, reducing overall processing time while maintaining accuracy through targeted execution of meaningful operations.
Solution Approach 2:
The system changes the parameter representation from dense to sparse formats. By representing neural network parameters as sparse tensors with explicit storage of only non-zero values and their locations, the system reduces the computational footprint and enables faster processing while maintaining the functional equivalence to dense networks, thus improving productivity without sacrificing reliability.
4Quantity of substance
If sparse neural networks are processed with conventional hardware, then computational resources are reduced, but processing speed decreases due to extensive operations on active values
Solution Approach 1:
The AI accelerator applies local quality optimization by designing specialized circuitry tailored to the specific characteristics of sparse tensor operations. The sparsity processing circuit and multiply circuits are locally optimized to handle the distribution patterns of active values in sparse tensors, enabling fast processing of computational resources without requiring conventional hardware.
Solution Approach 2:
The system performs preliminary actions by pre-processing sparse tensors to identify and organize active values before computation. The sparsity processing circuit determines locations of active values and reorganizes them into formats optimized for the multiply circuits, enabling faster execution by preparing data in advance and eliminating unnecessary operations during the main computation phase.
Data Source
AI summary
A hardware accelerator that is efficient at performing computations related to a sparse neural network. The sparse neural network may be associated with a plurality of nodes. One of the nodes includes one or more sparse tensors. The accelerator may compress the sparse tensor to a dense tensor. The sparse tensor may also be structured so that the dense locations in the tensor are blocked or partitioned. The accelerator may transpose the weight tensor and align the partitions of the tensor with the hardware architecture. The structured tensor has a balanced number of active values so that the active values can be processed by an efficient number of operating cycles of the accelerator. The accelerator may also perform bitwise and operation to determine the location of dense pairs in two sparse tensors to reduce the number of computations.