Block-Based Correlation Volume Sampling for Optical Flow
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing optical flow estimation techniques face a tradeoff between computational complexity and memory usage, with all-pairs correlation volume sampling consuming excessive memory and on-demand sampling leading to worse runtime performance, especially for high-resolution videos.
Innovation Solution
Generate block-based orderings for source and target images to create a sparse correlation volume, computing and caching correlation values for relevant pixel pairs, and iteratively refine flow vectors using machine learning models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all-pairs correlation volume sampling is used, then optical flow estimation accuracy is improved, but memory consumption increases quadratically with the number of pixels
Solution Approach 1:
The patent divides the image into multiple blocks and computes correlation volumes only for relevant block pairs identified through coarse-to-fine flow estimation. This segmentation approach reduces the correlation volume from O(N²) for all-pairs to a manageable subset, enabling high-resolution video processing while maintaining accuracy.
Solution Approach 2:
The patent applies different levels of correlation volume computation to different regions based on their importance. Regions with significant motion or occlusion receive more detailed correlation analysis, while static regions use coarser estimates, optimizing the balance between accuracy and memory usage.
2Quantity of substance
If on-demand sampling is used to reduce memory usage, then memory consumption is reduced, but runtime performance deteriorates due to frequent re-computation and irregular memory access
Solution Approach 1:
The patent performs preliminary coarse-to-fine flow estimation to identify relevant block pairs before computing the final correlation volume. This preliminary action enables subsequent fine-grained correlation computation to focus only on necessary regions, avoiding both the memory overhead of all-pairs and the re-computation costs of pure on-demand approaches.
Solution Approach 2:
The patent maintains continuous refinement of flow estimates through iterative correlation volume computation and sampling. Each iteration builds upon previous results, continuously improving flow accuracy without requiring complete re-computation, thus maintaining high runtime performance.
3Measurement precision
If correlation volume computation is performed for high-resolution video, then optical flow estimation accuracy is improved, but both memory and compute requirements become prohibitive
Solution Approach 1:
The patent segments high-resolution images into blocks and processes them through a coarse-to-fine pyramid structure. This segmentation reduces the computational complexity from O(N²) to a manageable level while preserving the ability to capture fine-grained motion details in the final refinement stage.
Solution Approach 2:
The patent introduces a temporal dimension through iterative refinement and a spatial hierarchy through block-based processing. This multi-dimensional approach transforms the intractable all-pairs problem into a sequence of manageable sub-problems that can be solved efficiently on available hardware.
Data Source
AI summary
One embodiment of the present invention sets forth a technique for performing optical flow estimation. The technique includes generating (i) a first block-based ordering of a first set of features associated with a source image and (ii) a second block-based ordering of a second set of features associated with a target image. The technique also includes matching a plurality of mappings between source pixels in the source image and target regions in the target image to a subset of blocks included in a correlation volume associated with the first block-based ordering and the second block-based ordering. The technique further includes computing a plurality of correlation values included in the subset of blocks and determining a plurality of flow vectors between the source image and the target image based on the plurality of correlation values.