Multi-View Stereo Depth Estimation via Learned Triangulation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional VR/AR/MR systems fail to provide a comfortable, rich, binocular 3D experience due to the vergence-accommodation conflict, and existing depth estimation methods are either costly, inefficient, or reliant on active sensors, which are not suitable for lightweight, low-cost, and power-efficient XR systems.
Innovation Solution
The method involves estimating depths from multi-view images using a shared RGB encoder and descriptor decoder, which outputs descriptors, detects interest points, and triangulates them using singular value decomposition to generate 3D points, followed by a sparse depth encoder and depth decoder to produce a dense depth image, reducing reliance on costly active sensors and improving computational efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional depth estimation methods are used, then depth accuracy may be maintained, but computational cost and system complexity increase significantly
Solution Approach 1:
The patent extracts and removes the need for expensive active depth sensors (LiDAR, structured light, time-of-flight cameras) from the system. Instead, it uses only passive RGB cameras to capture multi-view images, and employs a learned triangulation network to estimate depth from these images, thereby eliminating complex and costly sensing hardware while maintaining depth estimation capability
Solution Approach 2:
The patent replaces traditional mechanical/optical active sensing systems with a computational approach. Instead of using physical sensors to actively measure depth, the system uses a neural network that processes multi-view RGB images and learns to triangulate 3D points, substituting physical measurement mechanisms with learned computational models
2Measurement precision
If active sensors are used for depth estimation, then measurement accuracy is improved, but power consumption and cost increase
Solution Approach 1:
The patent replaces expensive, power-hungry active sensors with inexpensive passive RGB cameras. The system uses multiple standard RGB cameras that consume minimal power compared to active sensors like LiDAR or time-of-flight cameras, achieving cost-effective and energy-efficient depth estimation through computational methods rather than expensive hardware
3Productivity
If sparse point triangulation is used, then computational efficiency is improved, but depth map completeness deteriorates
Solution Approach 1:
The patent performs preliminary dense sampling of 3D points along epipolar lines before applying the triangulation network. By pre-generating a dense set of candidate points and then learning to select and refine them, the system achieves both computational efficiency and depth map completeness, avoiding the need to process all possible points while maintaining full coverage
Solution Approach 2:
The patent transitions from sparse point matching in traditional MVS to a continuous 3D space representation. By sampling points densely along epipolar lines and using a neural network to operate in this continuous space, the system achieves complete depth maps while maintaining computational efficiency through learned selection and refinement processes
4Measurement precision
If learned triangulation is applied, then depth estimation accuracy is improved, but computational complexity increases
Solution Approach 1:
The patent segments the depth estimation process into distinct modular components: multi-view image acquisition, interest point detection, descriptor matching, epipolar line sampling, triangulation network processing, and depth map generation. This segmentation allows each component to be optimized independently and facilitates efficient implementation through parallel processing and specialized hardware acceleration
Data Source
AI summary
Systems and methods for estimating depths of features in a scene or environment surrounding a user of a spatial computing system, such as a virtual reality, augmented reality or mixed reality (collectively, cross reality) system, in an end-to-end process. The estimated depths can be utilized by a spatial computing system, for example, to provide an accurate and effective 3D cross reality experience.