Sparse Vision Transformer Attention for Lower MHSA Complexity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Vision Transformers (ViTs) face challenges in scaling to large-scale vision tasks due to the quadratic computational complexity of multi-head self-attention (MHSA), particularly when processing high-resolution and high-dimensionality inputs, and existing methods like token pruning and fixed sparse attention patterns are less effective in capturing instance-dependent semantic information.
Innovation Solution
The development of a lightweight connectivity predictor module to learn instance-dependent attention patterns, using a connectivity score to estimate the relevance between tokens, resulting in sparse attention mechanisms that reduce computational complexity while maintaining accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multi-head self-attention (MHSA) is used in Vision Transformers, then relationship measurement between tokens is achieved, but computational cost increases quadratically with the number of tokens
Solution Approach 1:
The patent segments the full attention matrix into multiple sparse attention patterns, where each pattern captures different types of relationships (e.g., local, global, hierarchical). This segmentation allows the model to measure token relationships with reduced computational cost by only computing attention for selected token pairs rather than all possible pairs.
Solution Approach 2:
The patent introduces dynamic sparse attention patterns that adapt to the input content. The attention sparsity pattern is not fixed but dynamically determined based on the input image characteristics, allowing the model to maintain measurement precision while reducing computational cost for different types of inputs.
2Productivity
If token pruning is applied to reduce the number of tokens, then computational complexity is reduced, but accuracy degrades quickly when pruning early layers
Solution Approach 1:
The patent applies dynamic token pruning where the pruning pattern adapts based on the input image and network depth. Tokens are selectively pruned in a dynamic manner rather than uniformly, preserving important tokens in early layers while reducing complexity in deeper layers, thus maintaining accuracy while improving efficiency.
Solution Approach 2:
The patent applies different pruning strategies to different layers and different tokens based on their local importance. Early layers use more conservative pruning to preserve accuracy, while later layers can afford more aggressive pruning. This local quality approach ensures that critical tokens are retained where they matter most.
3Productivity
If fixed sparse attention patterns are used in MHSA, then computational efficiency is improved, but the capacity to capture instance-dependent semantic information is limited
Solution Approach 1:
The patent transitions from fixed sparse attention patterns to dynamic sparse attention patterns that adapt to each input instance. The attention sparsity pattern is determined dynamically based on the input image content, allowing the model to capture instance-dependent semantic information while maintaining computational efficiency through sparsity.
Solution Approach 2:
The patent changes the attention pattern parameters dynamically based on input characteristics. Instead of using a fixed sparsity pattern, the model adjusts the attention pattern parameters (which tokens attend to and with what weight) based on the specific input image, enabling adaptability while maintaining efficiency through controlled sparsity.
4Manufacturing precision
If ViT is scaled to large number of tokens for high-resolution inputs, then vision task performance is improved, but quadratic computational complexity becomes prohibitive
Solution Approach 1:
The patent segments the computation of attention for large numbers of tokens by dividing the token set into groups that attend to each other sparsely. This segmentation allows high-resolution inputs to be processed by breaking down the quadratic complexity into manageable sparse computations, maintaining vision task accuracy while reducing overall computational complexity.
Solution Approach 2:
The patent employs dynamic sparsity patterns that adapt to the number of tokens in the input. For high-resolution inputs with many tokens, the model dynamically adjusts the attention sparsity to reduce computational complexity while preserving the necessary attention relationships for accurate vision task performance.
Data Source
AI summary
Vision Transformers (ViT) have shown their competitive advantages performance-wise compared to convolutional neural networks (CNNs) though they often come with high computational costs. Methods, systems and techniques herein learn instance-dependent attention patterns, utilizing a lightweight connectivity predictor module to estimate a connectivity score of each pair of tokens. Intuitively, two tokens have high connectivity scores if the features are considered relevant either spatially or semantically. As each token only attends to a small number of other tokens, the binarized connectivity masks are often very sparse by nature providing an opportunity to accelerate the network via sparse computations. Equipped with the learned unstructured attention pattern, sparse attention ViT produces a superior Pareto-optimal trade-off between FLOPs and top-1 accuracy on ImageNet compared to token sparsity (48%˜69% FLOPs reduction of MHSA; accuracy drop within 0.4%). Combining attention and token sparsity reduces VIT FLOPs by over 60%.