Multi-Window Vision Transformer Training via Token Masking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training high-resolution visual models, such as visual transformers, is cost-prohibitive and results in poor performance when tested with images of higher resolutions than those used during training.
Innovation Solution
The use of a multi-window training scheme where a transformer module is trained using N windows of tokens from high-resolution images, allowing the model to learn both local and global interactions, thereby reducing the effective overall resolution during training without compromising the actual resolution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If high-resolution images are used for training visual transformers, then model performance on high-resolution images is improved, but training cost increases significantly
Solution Approach 1:
The training image is divided into multiple windows, and only a subset of these windows is selected for training the visual transformer. This segmentation approach allows the model to learn from high-resolution image regions without processing the entire high-resolution image, thereby reducing training cost while maintaining model performance on high-resolution images.
2Productivity
If the number of tokens is reduced during training, then training efficiency is improved, but information loss occurs
Solution Approach 1:
Different regions of the image are treated differently by selecting specific windows for training. The method focuses computational resources on locally important regions (the selected windows) while masking out other regions, thereby maintaining training efficiency while preserving critical information needed for high-resolution image understanding.
Data Source
AI summary
A training system includes: a transformer module having the transformer architecture and configured to perform a vision task; and a training module configured to: receive a training image having a predetermined resolution; determine N windows of tokens of pixels in the training image and mask the tokens of all of the other pixels of the training image that are outside of the N windows, where N is an integer greater than or equal to 2; input the N windows of tokens to the transformer module; train the transformer module based on an output of the transformer module generated based on the N windows of tokens; and test the transformer module using a test image having the predetermined resolution.