Transformer Attention Sampling for Image Processing Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transformer-based technology, originally developed for Natural Language Processing, faces challenges when extended to image-processing domains due to its high data-intensiveness and complexity of data items like images, which can overwhelm computing resources.
Innovation Solution
A modified transformer architecture that includes a modified attention component, which generates original attention information and then produces modified attention information by removing attention values based on score information generated from a classification token, reducing data-intensiveness and enabling processing on devices with limited resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If transformer-based technology is applied to process images and other complex data items, then the capability to handle non-NLP fields is improved, but the computational resources required become excessive
Solution Approach 1:
The patent extracts and removes less important attention values from the attention map based on score thresholds. This extraction principle directly reduces the data volume processed in subsequent transformer operations, thereby lowering computational resource requirements while preserving the essential information needed for accurate image classification.
Solution Approach 2:
Instead of processing all attention values equally, the patent applies partial action by selectively retaining only those attention values that exceed a certain score threshold. This partial processing approach maintains classification accuracy for important features while avoiding the computational overhead of processing all attention values, thus resolving the contradiction between versatility and resource consumption.
2Reliability
If all attention values are processed in the transformer, then classification accuracy is maintained, but the data-intensiveness increases computational cost
Solution Approach 1:
The patent extracts only the significant attention values that contribute meaningfully to classification accuracy. By removing redundant or low-importance attention values through threshold-based filtering, the system maintains reliable classification performance while significantly improving computational efficiency and reducing data-intensiveness.
Solution Approach 2:
The patent applies different quality levels to different attention values based on their importance scores. High-scoring attention values are preserved with full detail to maintain classification accuracy, while low-scoring values are removed or downsampled. This local differentiation optimizes the balance between reliability and productivity by allocating computational resources proportionally to their actual value.
Data Source
AI summary
A transformer is described herein for using transformer-based technology to process data items (e.g., image items). The transformer increases the efficiency of the transformer-based technology by using a modified attention component. In operation, the modified attention component accepts embedding vectors that represent a plurality of item tokens, together with a classification token. A first stage of the modified attention component generates original attention information based on the embedding vectors. A second stage generates score information based on a portion of the original attention information that pertains to the classification token. A third stage produces modified attention information by removing attention values from the original attention information, as guided by a sampling operation that is performed on the score information. The second and third stages do not rely on machine-trained values, which expedites the deployment of these functions in existing transformers.