Convolution Lowering Hardware for Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning neural networks, particularly convolutional neural networks, face performance bottlenecks due to computationally intensive operations, even with specialized processing hardware, as traditional convolution operations are less efficient than matrix multiplication operations and require significant memory resources for buffering.
Innovation Solution
Implementing on-the-fly convolution lowering in hardware accelerators by linearizing convolution layers into general matrix-to-matrix multiplication (GEMM) operations, utilizing GEMM units, and employing caching and blocking schemes to reduce memory access and optimize processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional convolution operations are used in neural networks, then the operations can be performed with standard processing hardware, but the computational efficiency is low and memory resources are excessively consumed
Solution Approach 1:
The patent transforms the convolution operation parameters by converting 2D spatial convolutions into 1D temporal convolutions through time-major layout reorganization. This parameter transformation enables the use of temporal convolution algorithms that are more efficient on modern processors, reducing both computational complexity and memory requirements while maintaining the same mathematical operation results.
Solution Approach 2:
The patent replaces the traditional mechanical convolution computation mechanism with a matrix multiplication mechanism. By reformulating convolution as matrix multiplication operations, the system can leverage highly optimized BLAS (Basic Linear Algebra Subprograms) libraries and GPU/TPU accelerators designed for matrix operations, achieving significant performance improvements over direct convolution implementation.
2Productivity
If specialized processing hardware is used for deep learning, then computational performance improves, but bottleneck effects slow down overall system performance
Solution Approach 1:
The patent performs preliminary data reorganization and layout transformation before data enters the specialized processing hardware. By pre-converting spatial data to time-major layout and preparing matrices in the appropriate format, the system eliminates bottlenecks at the hardware interface, allowing specialized units to operate at full throughput without waiting for data reformatting during computation.
Solution Approach 2:
The patent segments the convolution operation into distinct phases: data reorganization phase, matrix multiplication phase, and result reconstruction phase. This segmentation allows each phase to be optimized independently and executed by different hardware components simultaneously, preventing any single bottleneck from limiting overall system performance.
3Reliability
If convolution operations are performed with full buffering of input and filter maps, then complete computation can be performed, but memory usage increases significantly
Solution Approach 1:
The patent extracts and processes only the necessary portions of input and filter maps at each computation step rather than buffering entire maps. By using the time-major layout and incremental matrix multiplication approach, the system loads and processes data in smaller chunks, completing the full convolution computation while using minimal memory at any given time.
Data Source
AI summary
A disclosed computing system may include a special-purpose hardware device having an input subsystem, a linearization subsystem, and a matrix multiplication unit. The input subsystem may facilitate on-the-fly convolution lowering within a neural network convolution layer by directing input volume patches to logical unit(s) of the device. The linearization subsystem may be configured to receive a patch from the input subsystem and to linearize the patch by arranging elements of the patch as a portion of a data matrix row. The matrix multiplication unit of device may be configured to receive the data matrix from the linearization subsystem and to apply a filter matrix to the data matrix via a matrix multiplication operation. Various other methods, systems, and computer-readable media are also disclosed.