Unified Optical Flow and Depth Estimation With Gated Cost Volumes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The computational complexity of image processing systems, particularly for depth sensing and optical flow analysis, imposes significant power and resource demands, limiting the efficiency and performance of devices such as autonomous vehicles and extended reality devices, especially in mobile and smaller form factors.
Innovation Solution
A unified approach for simultaneous optical flow and depth estimation using neural networks that correlate feature sets across viewpoints and time periods, employing gated recurrent units and convolution operations to generate and merge intermediate information, reducing computational complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If separate neural networks are used for optical flow and depth estimation, then accuracy is improved, but device complexity and computational complexity increase
Solution Approach 1:
The patent merges separate optical flow and depth estimation neural networks into a unified model that processes both tasks simultaneously. The unified model shares common feature extraction layers and uses a single network architecture to perform both optical flow and depth estimation, reducing device complexity while maintaining accuracy through joint training and shared representations.
Solution Approach 2:
The unified neural network model performs multiple functions - both optical flow estimation and depth estimation - within a single architecture. The model is designed to handle both tasks concurrently, making the system more versatile and reducing the need for separate specialized networks, thereby lowering device complexity.
2Measurement precision
If separate neural networks are used for optical flow and depth estimation, then accuracy is improved, but power consumption increases
Solution Approach 1:
By merging separate neural networks into a unified model, the patent reduces the total computational load and power consumption. The unified model processes both optical flow and depth estimation in a single pass through the network, eliminating redundant computations that would occur with separate networks, thereby reducing power consumption while maintaining accuracy.
Solution Approach 2:
The unified model enables continuous processing of both optical flow and depth estimation tasks simultaneously without the need for separate processing passes. This continuous action approach reduces the overall computational overhead and power consumption compared to sequential or separate processing of the two tasks.
3Measurement precision
If separate neural networks are used for optical flow and depth estimation, then accuracy is improved, but resource demands increase
Solution Approach 1:
The patent combines separate neural networks into a unified model that shares computational resources, memory, and processing units. This merging reduces the total quantity of resources required - including model parameters, computational operations, and memory bandwidth - while maintaining the accuracy of both optical flow and depth estimation through joint processing.
4Device complexity
If computational complexity is reduced for mobile devices, then device complexity and power consumption decrease, but processing accuracy may be compromised
Solution Approach 1:
The unified model reduces device complexity by consolidating multiple networks into one, while maintaining processing accuracy through joint training and shared feature representations. The model is designed to achieve both goals simultaneously by leveraging the synergistic relationship between optical flow and depth estimation tasks.
Solution Approach 2:
The patent optimizes model parameters and architecture specifically for mobile devices with limited computational resources. By adjusting parameters such as network depth, width, and computational precision, the model achieves accurate processing on resource-constrained devices without compromising accuracy, balancing device complexity and processing performance.
Data Source
AI summary
Techniques and systems are provided for image processing. For instance, a process can include correlating a first set of features from a first viewpoint with a second set of features from a second viewpoint at a first time period to generate a first disparity cost volume; correlating a third set of features from the first viewpoint at a second time period with the first set of features to generate a first optical flow cost volume; gating the first disparity cost volume to generate first intermediate disparity information; gating the first optical flow cost volume to generate first intermediate optical flow information; correlating the first set of features, the second set of features, and the first intermediate optical flow information to generate disparity information for output; and correlating the third set of features, the first set of features, and the first intermediate disparity information to generate optical flow information for output.