Direct Convolution via Tiled GEMM Accelerators
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep learning hardware accelerators face inefficiencies in memory footprint and data movement traffic due to the im2col transformation required for convolution operations, which limits performance and energy efficiency.
Innovation Solution
The proposed solution involves direct convolution operations using tiled GEMM accelerators, which include peer-to-peer data transfer, parametrized data transfer, and data transformation to reduce memory traffic and footprint, enabling native convolution performance without extra transformations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If im2col transformation is used to enable convolution operations on GEMM accelerators, then convolution can be performed using existing hardware, but memory footprint and data movement traffic increase significantly
Solution Approach 1:
The patent segments the convolution operation into distinct phases: input activation loading, weight loading, convolution computation, and output writing. By segmenting the operation, the system can optimize data movement for each phase separately, loading only necessary data into on-chip buffers during computation rather than transforming entire input volumes through im2col.
Solution Approach 2:
The patent changes the computational dimensionality by performing convolutions directly in the spatial domain rather than transforming to column-major format. The GEMM accelerator computes convolutions by treating filter applications as matrix multiplications, where the input activation matrix is multiplied by weight matrices to produce output feature maps, eliminating the need for im2col transformation.
2Adaptability or versatility
If im2col transformation is used to enable convolution operations on GEMM accelerators, then convolution can be performed using existing hardware, but data movement traffic increases significantly
Solution Approach 1:
The patent performs preliminary loading of input activation data and weight data into on-chip buffers before computation begins. Input activations are loaded into input buffers and weights are loaded into weight buffers in advance, so that during the convolution computation phase, no additional data movement is required. This preliminary action eliminates the repeated data movement that would occur with im2col transformation.
Solution Approach 2:
The patent maintains continuous computation by keeping all necessary data in on-chip buffers throughout the convolution operation. Once input activations and weights are loaded, the GEMM accelerator continuously performs matrix multiplications to compute output feature maps without requiring additional data movement from off-chip memory, ensuring continuous useful action without interruption for data loading.
3Productivity
If dedicated convolution hardware units are used, then convolution operations are accelerated, but the system cannot support other DNN types without algorithm modification
Solution Approach 1:
The patent makes the GEMM accelerator universal by demonstrating that convolutions, fully connected layers, and other DNN operations can all be expressed as matrix multiplications. The same hardware unit that performs GEMM operations can accelerate convolutions by treating them as matrix multiplications between input activations and weight matrices, eliminating the need for dedicated convolution hardware while maintaining acceleration capabilities.
Solution Approach 2:
Instead of transforming convolutions into GEMM operations through im2col (conventional approach), the patent inverts the approach by directly using GEMM operations to compute convolutions in their native form. The input activation matrix is multiplied by weight matrices to produce output feature maps, computing convolutions directly without intermediate transformation, thereby supporting multiple DNN types with a single hardware architecture.
Data Source
AI summary
Systems and methods for efficient convolution based on matrix multiply and add (MMA) are described. An example processor having a plurality of processing lanes is configured to perform convolution of a matrix of activation elements and a filter matrix in accordance with a configurable series of instructions including a plurality of MMA instructions and shift instructions while reusing activation elements already loaded to the processor or associated memory over a plurality of MMA operations. The filter elements are held stationary at inputs to the processor for multiple cycles for multiple MMA operations while activations are streamed in. Associated methods are also described.