Multi-Axis Vision Transformer With Linear Global-Local Attention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The scalability of self-attention mechanisms in vision tasks is limited by the lack of efficient incorporation of global and local interactions, leading to computational inefficiencies and suboptimal performance on large datasets.
Innovation Solution
A multi-axis self-attention model is introduced, comprising local and global attention operations arranged in series, allowing for both local and global spatial interactions with linear complexity, and integrated with convolutional layers to form a hierarchical vision backbone.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If full self-attention is applied to capture global interactions, then model capacity and generalizability are improved, but computational complexity increases quadratically
Solution Approach 1:
The patent segments the self-attention mechanism into two distinct processing paths: a local attention path for capturing local spatial interactions and a global attention path for capturing global interactions. This segmentation allows each path to operate with appropriate complexity, avoiding the quadratic complexity of full self-attention while maintaining both local and global modeling capabilities.
Solution Approach 2:
The patent introduces a hierarchical structure with multiple resolution levels, transitioning from local to global attention across different spatial dimensions. By organizing attention operations in a hierarchical manner rather than applying full attention uniformly, the model achieves global interactions with linear complexity through the relationship between local and global processing stages.
2Productivity
If window-based attention is used to reduce computational complexity, then scalability is improved, but model capacity is limited due to loss of non-locality
Solution Approach 1:
The patent merges local attention and global attention mechanisms into a unified multi-axis self-attention block. The local attention path processes features at fine-grained spatial locations while the global attention path captures long-range dependencies, and both paths operate in parallel within the same computational framework, achieving both scalability and model capacity.
Solution Approach 2:
The patent resolves the trade-off between local and global attention by operating at multiple resolution dimensions simultaneously. The hierarchical architecture allows the model to process both local details and global context at different levels of abstraction, maintaining non-locality information while achieving linear scalability through the structured relationship between local and global processing stages.
3Adaptability or versatility
If hierarchical architecture is introduced to regularize model capacity, then generalizability is improved, but computational heavy operations are required at early or high-resolution stages
Solution Approach 1:
The patent segments the hierarchical processing into stages where local attention is applied at higher resolutions for detailed feature extraction, and global attention is applied at lower resolutions for contextual understanding. This segmentation allows the model to achieve generalizability through hierarchical organization while reducing computational burden by applying appropriate attention mechanisms at appropriate resolution levels rather than requiring heavy operations throughout all stages.
Data Source
AI summary
Provided is an efficient and scalable attention model that can be referred to as multi-axis attention. Example implementations can include two aspects: blocked local and dilated global attention. These design choices allow global-local spatial interactions on arbitrary input resolutions with only linear complexity. The present disclosure also presents a new architectural element by effectively blending the proposed multi-axis attention model with convolutions. In addition, the present disclosure proposes a simple hierarchical vision backbone, example implementations of which can be referred to as MaxViT, by simply repeating the basic building block over multiple stages. Notably, MaxViT is able to “see” globally throughout the entire network, even in earlier, high-resolution stages.