Optical Flow Estimation via Axial Attention Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing optical flow estimation methods face challenges in balancing computation and memory resources with accuracy, particularly in terminal devices with limited resources, as they often rely on transformers that consume significant computation and memory and may not effectively capture global context information due to orthogonal attention features.
Innovation Solution
The method involves processing images separately using an image processing pass that extracts and fuses attention feature maps along both width-axial and height-axial directions, generating attention feature maps that incorporate row and column relationship information, and estimating optical flow based on these feature maps to improve accuracy and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If transformer-based optical flow estimation is used, then global context information can be captured, but computation and memory consumption increase significantly
Solution Approach 1:
The patent segments the attention computation by separating width-axial and height-axial processing into independent one-dimensional attention operations. This divides the original two-dimensional attention matrix computation into two sequential one-dimensional operations, reducing computational complexity while maintaining global context capture capability
Solution Approach 2:
The patent transforms the computation from two-dimensional spatial attention to two sequential one-dimensional attention operations along different axes. By changing the dimensional approach from simultaneous 2D processing to sequential 1D processing, it reduces memory requirements and computation while preserving the ability to model global dependencies
2Use of energy by moving object
If axially separated attention mechanism is used to reduce computation, then computation amount decreases, but global context information is lost due to orthogonal independent features
Solution Approach 1:
The patent merges the independently computed width-axial and height-axial attention feature maps through element-wise multiplication and addition operations. This combining step integrates the orthogonal features to reconstruct comprehensive global context information that would be lost if either axial processing was performed independently
Solution Approach 2:
The patent introduces intermediate fusion feature maps that serve as mediators between the width-axial and height-axial attention operations. These intermediate representations allow information from both axial directions to be combined and exchanged, enabling global context recovery without requiring full two-dimensional attention computation
Data Source
AI summary
A method of estimating an optical flow includes processing, using an image processing pass, a first image and a second image separately, and estimating the optical flow based on a second image attention feature map of the first image and a second image attention feature map of the second image. The processing using the image processing pass includes extracting a feature map by encoding an image, outputting a first image fusion attention feature map by fusing row relationship information of the image with the image feature map, outputting a first image attention feature map of the image based on the first image fusion attention feature map and the image feature map, outputting a second image fusion attention feature map by fusing column relationship information of the image with the first image attention feature map, and generating a second image attention feature map of the image based on the second image fusion attention feature map and the first image attention feature map.