Multi-Axis Vision Transformer for Linear Global-Local Attention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transformers in computer vision face challenges in scalability due to the lack of efficient incorporation of global and local interactions, leading to computational inefficiencies and limited model capacity, especially on large datasets.
Innovation Solution
A multi-axis self-attention model is introduced, comprising local and global attention operations arranged in series, allowing for both local and global spatial interactions with linear complexity, balancing model capacity and generalizability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full self-attention is used in Vision Transformer, then global interactions are captured, but computational complexity becomes quadratic and scalability is limited
Solution Approach 1:
The patent segments the self-attention mechanism into two distinct components: local attention that operates on small windows of patches to capture local interactions, and global attention that operates on a downsampled feature map to capture global interactions. This segmentation reduces the computational complexity from quadratic to linear while maintaining both local and global modeling capabilities.
Solution Approach 2:
The patent introduces a spatial downsampling dimension where global attention is applied on a coarser feature map obtained through pooling operations. This dimensional transformation allows the model to capture global interactions at reduced computational cost, effectively trading resolution for computational efficiency in the global attention pathway.
2Productivity
If local attention is used to reduce computational complexity, then scalability improves, but model capacity is reduced due to loss of non-locality
Solution Approach 1:
The patent merges local attention and global attention into a unified multi-axis attention block where both operations are applied in sequence to the same feature map. This combination allows the model to benefit from both local precision (through window-based attention) and global context (through downsampled attention), achieving linear scalability without sacrificing model capacity.
3Reliability
If extensive pre-training is used to regularize model capacity, then image recognition performance improves, but computational resources and data requirements increase
Solution Approach 1:
The patent applies different attention mechanisms to different spatial regions and resolutions: local window-based attention is applied at high resolution to capture fine-grained local patterns, while global attention is applied at downsampled resolutions to capture coarse-grained global context. This differentiated approach provides strong inductive bias that regularizes the model without requiring extensive pre-training data.
Data Source
AI summary
Provided is an efficient and scalable attention model that can be referred to as multi-axis attention. Example implementations can include two aspects: blocked local and dilated global attention. These design choices allow global-local spatial interactions on arbitrary input resolutions with only linear complexity. The present disclosure also presents a new architectural element by effectively blending the proposed multi-axis attention model with convolutions. In addition, the present disclosure proposes a simple hierarchical vision backbone, example implementations of which can be referred to as MaxViT, by simply repeating the basic building block over multiple stages. Notably, MaxViT is able to “see” globally throughout the entire network, even in earlier, high-resolution stages.