Monocular Video Depth Estimation Using Scene Flow for Stable Moving Objects
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Determining depth in monoscopic videos is challenging due to the difficulty in distinguishing between camera motion and object motion, leading to inconsistent and flickering depth estimates, especially when both the camera and objects are moving.
Innovation Solution
A hybrid approach using a pre-trained depth model fine-tuned with a scene flow model to generate geometrically and temporally consistent depth images by adjusting parameters based on depth and pixel flow loss functions, incorporating optical flow and scene flow to stabilize depth estimation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If triangulation is used to determine depth in monoscopic images, then depth can be estimated, but the depth estimates become inconsistent and flicker when camera or objects are moving
Solution Approach 1:
The patent employs feedback mechanisms through loss functions that compare depth estimates across time steps. The depth loss function measures discrepancies between current and previous depth estimates, and the pixel flow loss function measures discrepancies in projected pixel positions. These feedback signals are used to guide the optimization of depth parameters, ensuring temporal consistency and reducing flickering in the depth map.
Solution Approach 2:
The patent changes the parameters being optimized from static depth values to temporal sequences of depth values. By modeling depth as a function of time and using recurrent neural networks, the system captures temporal dependencies and adjusts depth estimates dynamically based on motion information, thereby achieving stable and consistent depth maps throughout the video sequence.
2Device complexity
If monoscopic imaging is used, then device complexity is reduced, but depth determination becomes difficult due to inability to distinguish camera motion from object motion
Solution Approach 1:
The patent introduces optical flow as an intermediary that captures motion information between consecutive frames. This optical flow field serves as a mediator that helps disambiguate whether apparent pixel displacements are caused by camera motion or object motion. By incorporating this intermediary motion representation, the system can more accurately estimate depth despite using a simple monoscopic camera.
Solution Approach 2:
The patent adds the temporal dimension to depth estimation by modeling depth as a function of time. Instead of estimating depth from a single static image, the system uses sequences of images and optimizes depth parameters across multiple time steps. This temporal dimension provides additional constraints that help resolve the ambiguity in monoscopic depth estimation.
3Ease of manufacture
If depth model is fine-tuned using only image data, then training is simpler, but temporal consistency of depth images deteriorates
Solution Approach 1:
The patent ensures continuity of useful action by using recurrent neural networks that process depth estimates sequentially across time steps. The network maintains hidden states that carry information about previous frames, ensuring that depth estimation is continuous and temporally consistent. This continuous processing approach prevents abrupt changes and flickering in the depth map while keeping the training process manageable.
Data Source
AI summary
A method includes determining, based on a first image, a first depth of a first pixel and, based on a second image, a second depth of a second pixel that corresponds to the first pixel. The method also includes determining a first 3D point based on the first depth and a second 3D point based on the second depth, and determining a scene flow between the first and second images. The method additionally includes determining an induced pixel position based on a post-flow 3D point representing the first 3D point displaced according to the scene flow, determining a flow loss value based on the induced pixel position and a position of the second pixel and a depth loss value based on the post-flow 3D point and the second 3D point, and adjusting the depth model or the scene flow model based on the flow and depth loss values.