Monocular Flow and Depth Estimation With Cross-Consistency Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional machine-learning-based methods struggle to simultaneously estimate optical flow, depth, and scene flow accurately from monocular images, as they often focus on individual tasks rather than solving them jointly, leading to suboptimal performance in robotics applications like autonomous vehicles.
Innovation Solution
A system that jointly trains neural networks to estimate optical flow, depth, and scene flow by using self-supervised photometric loss and imposing consistency loss between different estimates, along with mid-level fusion of encoded image and depth context features, to improve the accuracy of these estimates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional machine-learning-based methods are used to estimate optical flow, depth, and scene flow from monocular images, then the system can perform these estimation tasks, but the accuracy of the estimates is suboptimal due to focusing on individual tasks rather than solving them jointly
Solution Approach 1:
The patent merges three separate estimation tasks (optical flow, depth, and scene flow) into a single joint training framework. The system uses a unified loss function that combines photometric loss, consistency loss, and smoothness loss to simultaneously optimize all three estimates, allowing the neural networks to leverage shared information and improve overall accuracy while maintaining manageable complexity through modular architecture design
2Reliability
If separate neural networks are trained for individual tasks (optical flow, depth, scene flow), then the training process is simpler, but the performance in downstream tasks such as object detection and mapping is suboptimal
Solution Approach 1:
The patent implements a multi-functional training system where a single joint training framework simultaneously optimizes multiple estimation tasks. The unified loss function and shared neural network components enable the system to perform optical flow, depth, and scene flow estimation together, improving reliability for downstream tasks while managing complexity through efficient resource sharing and coordinated optimization across all tasks
Data Source
AI summary
Systems and methods described herein relate to jointly training a machine-learning-based monocular optical flow, depth, and scene flow estimator. One embodiment processes a pair of temporally adjacent monocular image frames using a first neural network structure to produce a first optical flow estimate; processes the pair of temporally adjacent monocular image frames using a second neural network structure to produce an estimated depth map and an estimated scene flow; processes the estimated depth map and the estimated scene flow using the second neural network structure to produce a second optical flow estimate; and imposes a consistency loss between the first optical flow estimate and the second optical flow estimate that minimizes a difference between the first optical flow estimate and the second optical flow estimate to improve performance of the first neural network structure in estimating optical flow and the second neural network structure in estimating depth and scene flow.