Low-Footprint Optical Flow and Stereo Matching With Streamable Attention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for optical flow (OF) and depth from stereo (DFS) estimation require large memory footprints, making them impractical for resource-constrained environments, and involve multi-pass softmax operations that create computation pipeline breaks and quadratic memory buffering.
Innovation Solution
A machine learning model incorporating a softmax with norm folding mechanism and a streamable attention mechanism that eliminates cost volumes and performs depth-first computation, reducing memory usage and eliminating the need for buffering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional cost volumes are used for OF and DFS estimation, then computation accuracy is maintained, but memory footprint becomes excessively large
Solution Approach 1:
The patent segments the computation process into distinct phases: feature extraction from input images, attention mechanism computation, and final OF/DFS estimation. By processing features and attention weights incrementally rather than computing full cost volumes, the method divides the large memory requirement into manageable segments that can be processed sequentially with minimal buffering.
Solution Approach 2:
The patent extracts only the essential features from input images using feature extractors, and then extracts attention weights through the attention mechanism. This extraction approach eliminates the need to store and process complete cost volumes, retaining only the critical information needed for accurate OF and DFS estimation while dramatically reducing memory footprint.
2Measurement precision
If multi-pass softmax operations are performed, then normalized probability values are accurately computed, but computation pipeline continuity is broken and latency increases
Solution Approach 1:
The patent performs preliminary computation of the sum of exponential values during the first pass through the softmax operation. This pre-computed sum is then reused in the second pass to normalize the probability values, eliminating the need to re-traverse the entire input tensor and maintaining computation pipeline continuity while ensuring accurate normalization.
Solution Approach 2:
The patent maintains continuous data flow through the computation pipeline by buffering only the essential intermediate values (exponential computations and their sum) rather than the entire input tensor. This allows the softmax operation to proceed in two passes without breaking pipeline continuity, as the buffered values are readily available for the normalization step.
3Measurement precision
If quadratic complexity attention mechanisms are used, then comprehensive interaction between image features is captured, but memory buffering requirements become prohibitive
Solution Approach 1:
The patent applies local quality by computing attention weights between specific pairs of features from left and right images rather than computing all possible interactions. The attention mechanism focuses computational resources on relevant feature pairs, capturing essential interactions while avoiding the quadratic memory buffering requirements of exhaustive attention computation.
4Quantity of substance
If GPUs with large on-chip memory are used, then cost volumes and intermediate computations can be stored, but device cost and power consumption increase
Solution Approach 1:
The patent changes the computational parameters by eliminating cost volume computation and using feature-based attention mechanisms instead. This parameter change reduces the memory capacity requirement from gigabytes to kilobytes, enabling deployment on resource-constrained devices with low power consumption while maintaining computation accuracy for OF and DFS estimation.
Data Source
AI summary
A device includes a memory configured to store input data, and also includes one or more processors configured to process the input data using a machine learning model that incorporates a softmax with norm folding mechanism.