Monocular Depth Mapping With Sparse Inputs to Resolve Scale Ambiguity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current depth sensing technologies for VR and AR applications are limited by high power consumption, limited range, and accuracy issues, particularly in outdoor environments, and monocular depth estimation suffers from scale ambiguity and high relative error, making them unsuitable for robust use in applications like 3D reconstruction and autonomous driving.
Innovation Solution
A deep neural network model that utilizes sparse depth inputs from various sources, such as sparse depth sensors or LiDAR, to produce high-quality, dense depth maps, adaptable to arbitrary sampling patterns, and capable of resolving scale ambiguity, achieving accuracy comparable to conventional depth sensors both indoors and outdoors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If LiDAR or time-of-flight depth sensors are used, then depth measurement accuracy is improved, but power consumption increases and device weight increases
Solution Approach 1:
The patent replaces active depth sensing hardware (LiDAR, time-of-flight sensors) with a computational approach using monocular depth estimation neural networks. The system processes standard RGB images through trained neural networks to generate depth maps, eliminating the need for power-intensive active sensing components while maintaining acceptable depth measurement accuracy for AR applications.
Solution Approach 2:
The patent creates a computational copy of depth information by generating synthetic depth maps from RGB images through neural network processing. Instead of physically measuring depth with sensors, the system creates a virtual representation of depth data that can be used for AR occlusion and spatial understanding without requiring additional sensing hardware.
2Device complexity
If monocular depth estimation is used, then device complexity is reduced, but scale ambiguity and relative error increase
Solution Approach 1:
The patent applies preliminary action by pre-training neural networks on large datasets of paired RGB and depth images before deployment. The networks are trained offline to learn the complex mapping between 2D images and 3D depth structures, so that during actual AR operation, the pre-trained models can quickly generate accurate depth estimates without requiring complex runtime processing or additional sensors.
3Area of stationary object
If dense depth maps are generated using conventional sensors, then depth coverage is improved, but power consumption and device weight increase
Solution Approach 1:
The patent makes the RGB camera universal by enabling it to serve dual functions: capturing color images for visual display and generating depth information through neural network processing. This eliminates the need for separate depth sensing hardware, allowing the same camera component to provide both photometric and geometric information for AR applications.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
Systems and methods are disclosed for training and using neural networks for computing depth maps based on a single image. One method Systems and methods are disclosed for training and using neural networks for computing depth maps. One method for training the neural network includes providing an image input to the neural network. The image input may include a camera image of a training scene. The method may also include providing a depth input to the neural network. The depth input may be based on a high-density depth map of the training scene and a sampling mask. The method may further include generating, using the neural network, a computed depth map of the training scene based on the image input and the depth input. The method may further include modifying the neural network based on an error between the computed depth map and the high-density depth map.