Sparse Neural Network Convolution on CPUs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network execution methods require specialized hardware accelerators like GPUs and TPUs for efficient computation, limiting the performance of commercially available CPUs in executing deep neural networks due to the assumption that highly parallelized computation is necessary.
Innovation Solution
Exploiting the natural sparsity in neural network computations, specifically through activation sparsity, to enable CPUs to perform competitively by compressing sparse input matrices and optimizing matrix multiplication operations, thereby reducing unnecessary computations and data movement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If specialized hardware accelerators (GPUs, TPUs) are used for neural network computation, then computational throughput and performance are improved, but device complexity and cost increase
Solution Approach 1:
The patent extracts and exploits the sparsity property inherent in neural network activations, separating the computation of zero elements from non-zero elements. By identifying and removing redundant zero computations, the system achieves efficient processing on standard CPUs without requiring specialized accelerator hardware, thus resolving the contradiction between throughput and complexity.
Solution Approach 2:
The patent changes the computational parameters by dynamically adjusting the execution strategy based on sparsity patterns. It transforms the computation from a dense matrix multiplication approach to a sparse-aware approach that adapts to the actual data characteristics, enabling CPUs to achieve accelerator-level performance for sparse workloads.
2Ease of manufacture
If dense matrix multiplication is performed without considering sparsity, then implementation simplicity is maintained, but computational efficiency and resource utilization deteriorate
Solution Approach 1:
The patent introduces dynamic computation by adjusting the execution strategy based on runtime sparsity patterns. The system dynamically identifies zero elements and skips their computation, transforming the static dense multiplication into an adaptive sparse-aware computation that maintains simplicity while dramatically improving efficiency.
Solution Approach 2:
The patent applies partial action by performing computation only on necessary non-zero elements rather than all elements in the matrix. This selective computation approach avoids the excessive action of processing zero elements, achieving high efficiency while maintaining implementation simplicity through straightforward conditional logic.
3Reliability
If all matrix elements are processed including zeros, then computational completeness is ensured, but unnecessary computations increase processing time and energy consumption
Solution Approach 1:
The patent converts the harmful effect of sparsity (irregular data patterns) into a benefit by exploiting zero elements to skip computations. The presence of zero elements, which would normally be processed wastefully, becomes the basis for optimization by identifying and eliminating redundant operations, thus reducing processing time while maintaining computational completeness for non-zero elements.
4Measurement precision
If full precision arithmetic is used for all computations, then numerical accuracy is maintained, but computational overhead and memory bandwidth usage increase
Solution Approach 1:
The patent extracts and processes only the necessary non-zero elements, separating them from the zero elements that would consume memory bandwidth. By loading and processing only the relevant data elements, the system maintains numerical accuracy for computations while dramatically reducing memory bandwidth usage and energy consumption associated with data movement.
Data Source
AI summary
A system and method of executing a convolution layer of a neural network may include: (a) selecting an output spatial position (OSP) of an output matrix data element of the convolution layer; (b) selecting, based on the selected OSP, a non-zero input element of an input matrix data element; (c) producing, based on the selected OSP, a vector of kernel elements from a kernel matrix data element; (d) performing a vectoral multiplication operation of the selected non-zero input element and the vector of kernel elements, and accumulating a product of the vectoral multiplication in a vector register of a processor; (e) repeating (c) and (d) with subsequent non-zero input elements and corresponding vectors of kernel elements to obtain an outcome of the convolution of the selected OSP; and (f) repeating (a) through (e) with subsequent selection of OSPs, to obtain an outcome of the convolution layer.