Mobile CNN Convolution Layout for Low-Latency Matrix Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional GEMM-based convolution operations in mobile environments suffer from inefficiencies such as data duplication and rearrangement overhead, leading to increased latency and reduced data reuse, which are not suitable for rapid response to individual user requests.
Innovation Solution
A low-latency convolution operation apparatus and method that includes a matrix operation unit performing element-wise matrix operations, matching input and filter matrices considering convolution parameters, generating output blocks by computing filter and input blocks in specific directions, and aggregating them into a single matrix, adding a new computation dimension for the filter spatial dimension to optimize CNN computations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If conventional GEMM operations are used for convolution in mobile environments, then data duplication and movement overhead increases, but processing speed for individual requests improves
Solution Approach 1:
The patent segments the convolution operation into depthwise convolution and pointwise convolution parts. The depthwise convolution processes each input channel independently using depthwise convolution kernels, reducing data duplication. The pointwise convolution then processes the depthwise output using 1x1 kernels. This segmentation allows mobile devices to perform convolution efficiently without full GEMM overhead.
Solution Approach 2:
The patent introduces a new computation dimension by processing convolution in the depthwise direction (spatial dimension) rather than traditional GEMM matrix multiplication. By iterating through the depthwise dimension and computing output blocks in this direction, the system achieves better data reuse patterns and reduces unnecessary data duplication while maintaining computational correctness.
2Productivity
If batch processing is used to improve data reuse rate, then average processing speed increases, but latency for individual requests increases
Solution Approach 1:
The patent implements dynamic processing where the system can adaptively choose between batched and unbatched processing modes. The depthwise convolution operation processes data in a dynamic manner that respects the temporal characteristics of mobile requests, allowing immediate response for individual requests while still benefiting from optimized data reuse patterns through the structured depthwise iteration.
3Ease of manufacture
If CNN is converted to GEMM using conventional Im2Col approach, then computation can be performed using matrix multiplication, but additional processing time is required for data duplication and rearrangement
Solution Approach 1:
The patent extracts the essential convolution operation from the full GEMM framework and implements it through depthwise convolution that processes input channels independently. This extraction eliminates the need for complex Im2Col data rearrangement while maintaining the ability to perform convolution using efficient kernel operations, achieving both computational compatibility and time efficiency.
Data Source
AI summary
A low-latency convolution operation apparatus according to an embodiment includes: a memory in which a matrix operation program is stored; and a processor configured to execute the program stored in the memory, wherein the matrix operation program includes a matrix operation unit configured to generate an output matrix by performing element-wise matrix operations between an input matrix and a filter matrix, the matrix operation unit being configured to: (a) match rows of the input matrix with columns of the filter matrix in consideration of convolution parameters including stride and dilation; (b) compute filter blocks that move in a depthwise direction and input blocks that move in a horizontal direction along the width dimension of a filter tensor to generate output blocks; and (c) repeat steps (a) and (b) and aggregate all computed output blocks into a single matrix to generate the output matrix.