Vision Transformer Self-Attention Redundancy via Local Quality
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Feature extractors using Vision Transformer (ViT) often fail to learn effectively, leading to reduced robustness against position changes and object hiding in images, resulting in decreased performance due to Self-Attentions paying attention to the same partial areas.
Innovation Solution
An image processing apparatus that divides input images into partial images, converts them into tokens, adds class-tokens, and uses a Transformer Encoder to update tokens based on relevance, combining class-token encoded representations to produce a feature vector, ensuring Self-Attentions focus on different areas.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If ViT uses multiple Self-Attentions to aggregate features from partial areas, then the model can extract comprehensive image information, but the Self-Attentions may learn to pay attention to the same partial areas causing redundancy and decreasing performance
Solution Approach 1:
The patent applies local quality by assigning different spatial sampling patterns to different Self-Attention mechanisms. Specifically, odd-numbered Self-Attentions sample from odd-numbered partial images while even-numbered Self-Attentions sample from even-numbered partial images. This creates localized specialization where each Self-Attention focuses on specific spatial regions, eliminating redundancy and improving robustness against position changes and object hiding.
2Productivity
If ViT parameters are to be effectively unitized, then Self-Attentions should pay attention to different partial areas, but current learning processes cause Self-Attentions to focus on the same areas reducing parameter efficiency
Solution Approach 1:
The patent segments the feature extraction task across multiple Self-Attentions by dividing the image into partial images and assigning different sampling patterns to different Self-Attentions. Odd-numbered Self-Attentions process odd-numbered partial images while even-numbered ones process even-numbered partial images. This segmentation ensures each parameter set handles distinct spatial information, maximizing parameter utilization efficiency while maintaining performance robustness.
Data Source
AI summary
This disclosure provide an apparatus which comprises an acquisition unit that acquires an image, a divider that divides the image into a plurality of partial images; a converter that converts the partial images into tokens having fixed dimensional vectors; an adder that adds a class-token having a vector of a dimension same as the two or more tokens to the tokens obtained by the converter; an encoder that updates the tokens obtained by the adder based on a relevance between the tokens to obtain final encoded representations; an acquiring unit that acquires an encoded representation corresponding to the class-token in the encoded representations obtained by the encoder as class-token encoded representations; and a combining unit that combines the class-token encoded representations to obtain a feature vector of the input image.