Vision Transformer Fusion Block Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing vision transformer technologies for mobile devices are heavy in weight, limiting their scalability and performance in computer vision tasks such as image classification, object detection, and segmentation, despite their ability to learn global representations.
Innovation Solution
An optimized mobile vision transformer block is introduced, combining convolutional neural networks and vision transformers, utilizing a depthwise-separable convolutional layer and a fusion block with a 1×1 convolutional layer to reduce parameters and enhance feature extraction, allowing for better scalability and performance in computer vision tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If vision transformers are used to learn global representations, then image classification and object detection performance is improved, but the model weight increases making it heavy and less scalable
Solution Approach 1:
The patent divides the image into non-overlapping patches and processes them through separate convolutional branches (local and global representation blocks) before fusion, enabling efficient feature extraction with reduced parameters compared to full attention mechanisms
Solution Approach 2:
The patent combines convolutional neural network efficiency with vision transformer global representation capabilities by merging local and global feature representations through a fusion block, achieving both performance and parameter efficiency
2Adaptability or versatility
If standard convolutional layers are used in vision transformers, then feature extraction capability is maintained, but the number of parameters and computational cost increase
Solution Approach 1:
The patent applies different convolutional operations to different parts of the feature extraction process: depthwise separable convolutions in the local representation block for efficient local features, and standard convolutions in the global representation block for global context, optimizing the balance between capability and complexity
Solution Approach 2:
The patent changes the parameter efficiency by using depthwise separable convolutions instead of standard convolutions in specific blocks, reducing the number of parameters while maintaining feature extraction capability through architectural design
Data Source
AI summary
A system for optimizing a vision transformer block for use with mobile vision transformers utilized for tasks, such as image classification, segmentation, and objected detection is disclosed. The system includes incorporating a 1×1 convolutional layer in place of a 3×3 convolutional layer in a fusion block of the vision transformer block to reduce constraints on scaling neural network size. Additionally, the system includes fusing local and global representations in the fusion block of the vision transformer block instead of fusing input features and global representations. Furthermore, the system includes fusing input features in the fusion block by adding the input features to the output of the 1×1 convolutional layer of the fusion block. Moreover, the system includes substituting a 3×3 convolutional layer in the local representation block of the vision transformer block with a depthwise-separable 3×3 convolutional layer. The optimized transformer block enhances image classification, segmentation, and object detection.