Depth Prediction From RGB and Geometry for Missing Sensor Data

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional depth sensors, such as RGB-D cameras and LiDAR, often fail to capture accurate depth values for transparent objects and dark portions of images due to lack of light reflection or occlusion, leading to incomplete depth data in image processing applications.

Innovation Solution

A machine learning-based system that predicts missing depth values by generating ray-voxel pairs, using a Local Implicit Depth Function (LIDF) to estimate depth and refine predictions through a refinement model, combining color and geometric data to infer depth information even in areas where sensors fail.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If conventional depth sensors (RGB-D cameras, LiDAR) are used to capture depth data, then depth information can be obtained for most objects, but transparent objects and dark portions fail to produce correct depth values due to lack of light reflection

Engineering Contradiction:

Improvedepth data accuracyVSAvoidmissing depth values

Core Design Contradiction:

ReliabilityVSLoss of information

Solution Approach 1:

The patent uses RGB color information as an intermediary to infer depth values for transparent objects. Instead of relying solely on direct depth sensor measurements which fail for transparent objects, the system uses the color data from RGB cameras as a mediator to predict and fill in missing depth information through machine learning models.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent replaces the mechanical/optical depth sensing mechanism with a computational approach. Instead of using physical light reflection and time-of-flight measurements that fail for transparent objects, the system substitutes a machine learning-based prediction system that processes RGB and existing depth data to generate depth values for transparent objects.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Loss of information

If machine learning techniques are used to predict missing depth values, then depth information for transparent objects can be recovered, but computational complexity and processing time increase

Engineering Contradiction:

Improvemissing depth valuesVSAvoidprocessing complexity

Core Design Contradiction:

Loss of informationVSDevice complexity

Solution Approach 1:

The patent performs preliminary actions by pre-processing RGB and depth data into feature representations before depth prediction. The system extracts relevant features from color and existing depth information in advance, which simplifies the subsequent depth prediction task and reduces real-time computational complexity.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent transforms the depth prediction problem by changing parameters from direct coordinate prediction to probability distribution prediction. The machine learning model predicts depth values as probability distributions over discrete depth bins, which simplifies the optimization landscape and enables more efficient computation compared to continuous parameter optimization.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS11798183B2Machine learning techniques for predicting depth information in image data

Publication Date: 2023.10.24 NVIDIA CORP

AI summary

Apparatuses, systems, and techniques to estimate or predict depth information for image data. In at least one embodiment, depth information is predicted based at least in part on color information and geometry information associated with an image.