Separable Convolution on Matrix Processing Arrays
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Matrix processing systolic arrays in AI accelerators face performance, complexity, and power consumption issues due to their limited ability to efficiently support image processing filters, which require shifting filters by a single element in each time step across large input images, making caching and data reuse impractical.
Innovation Solution
Implementing separable convolution filter operations on matrix multiplication arrays by decomposing 2D matrix operations into vector instructions and adding unidirectional cross-channel communication between FMA units, allowing for efficient execution of image processing filters through the use of NxN kernels decomposed into Nx1 and 1xN vectors, reducing the number of loads and arithmetic operations by approximately 50%.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If simple GEMM operations are supported on matrix processing arrays, then performance is improved, but the ability to efficiently support image processing filters deteriorates
Solution Approach 1:
The patent segments the 2D convolution operation into two separate 1D operations: first applying a vertical filter kernel to rows of the input image, then applying a horizontal filter kernel to the intermediate results. This segmentation allows the matrix processing array to efficiently handle image processing filters by breaking down complex 2D operations into simpler 1D operations that can be executed using standard GEMM operations.
2Ease of operation
If filters are shifted by a single element in each time step across large input images, then image processing is performed, but caching and data reuse become inefficient
Solution Approach 1:
By segmenting the 2D convolution into two 1D passes, the patent enables each pass to process entire rows or columns at once rather than shifting filters element-by-element. This allows for more effective data reuse within each pass, as the same filter kernel is applied repeatedly to different rows or columns without requiring frequent cache updates.
Solution Approach 2:
The patent maintains continuous useful action by processing entire rows or columns in each pass without interruption. The matrix processing array continuously applies the filter kernel across multiple elements in parallel, maximizing data reuse and minimizing cache access overhead compared to sequential single-element shifting.
3Ease of operation
If 2D convolution operations are performed directly, then image processing is achieved, but the number of loads and arithmetic operations increases
Solution Approach 1:
The patent reduces computational complexity by segmenting the 2D convolution operation into two separate 1D convolutions. Instead of performing a full 2D convolution with O(H×W×K×K) operations where H and W are image dimensions and K is kernel size, the segmented approach performs two 1D convolutions with O(H×W×K) operations each, reducing the total number of arithmetic operations by approximately 50%.
Data Source
AI summary
Methods and apparatus relating to separable convolution filter operations on matrix multiplication arrays are described. In an embodiment, logic circuitry generates a first convolution kernel and a second convolution kernel based on a two-dimensional convolution kernel. A matrix processing array comprising a plurality of Fused Multiply-Add (FMA) blocks applies the first convolution kernel to input data during a first pass to generate an intermediate data and the matrix processing array applies the second convolution kernel to the intermediate data to generate output data. Other embodiments are also disclosed and claimed.