3D Self-Attention Projections for Spatially Preserving Image Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network models for image recognition, particularly transformer models, struggle with maintaining both global and local information, leading to a loss of spatial information when applied to image recognition tasks.
Innovation Solution
Implement a self-attention mechanism that generates 3D query and key data through convolution operations, followed by vertical and horizontal projections to create 2D data, and then determines attention results through multiplications and normalizations, preserving spatial information while reducing computational complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If transformer models are applied to image recognition, then global information processing is improved, but spatial information is lost
Solution Approach 1:
The patent introduces a depth dimension to the attention mechanism by processing 3D feature maps (channel×height×width) instead of traditional 2D images. The self-attention operates on volumetric data, creating 3D query and key representations that preserve spatial relationships while enabling global context modeling through the added depth dimension.
Solution Approach 2:
The patent segments the attention computation into independent vertical and horizontal projection operations. The vertical projection processes channels while preserving spatial dimensions, and the horizontal projection processes spatial dimensions while preserving channels. This segmentation allows the model to maintain spatial information through separate processing streams while achieving global context through their combination.
2Loss of information
If self-attention mechanism is implemented with 3D data, then spatial information is preserved, but computational complexity increases
Solution Approach 1:
The patent divides the 3D self-attention computation into two independent 2D projection operations: vertical projection (processing channel dimension) and horizontal projection (processing spatial dimensions). This segmentation reduces the computational complexity from O((H×W)²) for full 3D self-attention to O(H² + W²) by processing dimensions separately, while still preserving spatial information through the combination of both projections.
3Device complexity
If projection operations are applied to reduce data dimensions, then computational overhead is reduced, but information loss may occur
Solution Approach 1:
The patent applies segmentation to the projection operations, where vertical projection processes only the channel dimension while horizontal projection processes only the spatial dimensions. This segmented approach reduces computational overhead by avoiding full 3D matrix operations, and preserves spatial information because the horizontal projection explicitly maintains spatial relationships through its processing of height and width dimensions.
Data Source
AI summary
A method with self-attention includes: obtaining a three-dimensional (3D) feature map; generating 3D query data and 3D key data by performing a convolution operation based on the 3D feature map; generating two-dimensional (2D) vertical data based on a vertical projection of the 3D query data and the 3D key data; generating 2D horizontal data based on a horizontal projection of the 3D query data and the 3D key data; determining an intermediate attention result through a multiplication based on the 2D vertical data and the 2D horizontal data; and determining a final attention result through a multiplication based on the intermediate attention result and the 3D feature map.