Vision Transformer Channel Splitting for Lower Edge Inference Cost
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Vision transformers require high computational costs and long latency due to multi-head self-attention, making them impractical for deployment on edge/mobile devices and laptops.
Innovation Solution
A novel architecture that splits image channels into portions processed with depthwise convolution and multi-head self-attention, reducing computational complexity by token and channel dimension reduction, and employs a window-shifting mechanism for channel selection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multi-head self-attention is used in vision transformers, then performance on computer vision tasks is improved, but computational cost and latency increase significantly
Solution Approach 1:
The patent segments the channel dimension into multiple portions, with different portions processed by different operations (depthwise convolution vs. multi-head self-attention). This segmentation allows the system to maintain performance benefits of attention mechanisms while reducing overall computational burden by applying expensive operations only to selected channel portions.
Solution Approach 2:
The patent applies different processing qualities to different parts of the input: depthwise convolution is applied to some channel portions while multi-head self-attention is applied to others. This local differentiation allows optimization of computational resources by applying expensive operations only where most beneficial.
2Measurement precision
If multi-head self-attention is applied to high resolution images, then performance is improved, but computational complexity becomes quadratic to input token size
Solution Approach 1:
The patent segments channels and processes only selected portions with multi-head self-attention, avoiding quadratic complexity growth on the full channel dimension. This segmentation strategy maintains performance on high-resolution images while controlling computational complexity.
Solution Approach 2:
The patent applies multi-head self-attention to only a portion of the channels rather than all channels, achieving sufficient performance through partial application of the expensive operation. This partial action approach avoids the full quadratic computational burden.
3Device complexity
If token size is reduced to decrease MHSA complexity, then computational cost is reduced, but performance may be degraded
Solution Approach 1:
Instead of reducing token size (spatial dimension), the patent reduces complexity by operating on channel portions. This dimensionality change allows complexity reduction without sacrificing spatial resolution and performance, as the channel dimension provides an alternative optimization target.
Data Source
AI summary
A computer-implemented method for vision transforming includes, for each of one or more channels of each of a set of tiles of an image: splitting the channel into at least a first channel portion and a second channel portion; processing the first channel portion using depthwise convolution; processing the second channel portion with multi-head self-attention; and combining the processed first channel portion and the processed second channel portion; and identifying an object in the image at least partially based on the combined processed first channel portion and processed second channel portion for each of the one or more channels of each of the set of tiles of the image.