Depth Map Reprojection for Temporally Consistent Stereo Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional mixed-reality systems face challenges in generating depth maps with high computational expense, latency, imprecise depth borders, and temporal inconsistencies, leading to parallax errors and artifacts in parallax-corrected images.
Innovation Solution
The system employs a method for temporally consistent depth map generation by obtaining a first stereo pair of images, generating a first depth map, and using a reprojected depth map to align with a second stereo pair of images, thereby improving temporal consistency and overall depth map quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional stereo matching algorithms are used to generate depth maps, then depth information can be obtained, but computational expense increases and latency occurs
Solution Approach 1:
The system performs preliminary actions by reprojecting the previous depth map to the current stereo pair's viewpoint before generating the new depth map. This preliminary reprojection creates a temporal reference that guides the current depth estimation, reducing the computational search space and accelerating the matching process while maintaining accuracy.
Solution Approach 2:
The system maintains continuity by incorporating the previous depth map into the current frame's depth estimation process. Instead of independently processing each frame, the depth information flows continuously across frames through reprojection and fusion, eliminating redundant computations and reducing latency while preserving measurement precision.
2Measurement precision
If conventional stereo matching algorithms are used to generate depth maps, then depth information can be obtained, but temporal inconsistencies occur leading to depth flickers
Solution Approach 1:
The system implements feedback by using the reprojected previous depth map as a reference guide for the current depth estimation. The algorithm compares current stereo disparities with the reprojected historical depth information and uses this feedback to constrain and stabilize the depth estimation, preventing temporal flickers while maintaining precision.
Solution Approach 2:
By reprojecting the previous depth map to the current viewpoint before generating the new depth map, the system establishes a temporal reference in advance. This preliminary action creates a stable baseline that guides the current estimation and ensures temporal consistency, preventing flickers caused by independent frame processing.
3Measurement precision
If depth maps are generated for each stereo pair, then depth information is available, but computational expense increases
Solution Approach 1:
The system merges the current stereo matching results with the reprojected previous depth map to generate the final depth map. This combination allows the algorithm to leverage historical depth information, reducing the computational effort required for current frame processing while maintaining high depth map quality through the fusion of multiple sources.
Solution Approach 2:
The reprojection of the previous depth map serves as a preliminary computation that prepares a reference solution in advance. This preliminary action reduces the computational burden of the current depth estimation by providing a starting point or constraint, thereby lowering overall computational expense while preserving depth map quality.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Systems and methods are provided for performing temporally consistent depth map generation by implementing acts of obtaining a first stereo pair of images of a scene associated with a first timepoint and a first pose, generating a first depth map of the scene based on the first stereo pair of images, obtaining a second stereo pair of images of the scene associated with at a second timepoint and a second pose, generating a reprojected first depth map by reprojecting the first depth map to align the first depth map with the second stereo pair of images, and generating a second depth map that corresponds to the second stereo pair of images using the reprojected first depth map.