GAN-Generated Raw Depth Maps for Realistic Depth Completion Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing depth sensors in industrial settings produce raw depth maps with large errors and loss of information due to optical reflections, deflections, and refractions, especially with reflective metallic objects, necessitating cumbersome and costly real-world data acquisition for training depth completion models.
Innovation Solution
Generate synthetic raw depth maps from CAD data using a generative adversarial network (GAN) to create training data for depth completion models, incorporating a marginal preservation loss to maintain geometric consistency and avoid object removal, and use these synthetic data alongside real raw depth maps for training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If real-world depth sensor measurements are used to acquire training data, then the training data reflects real-world conditions, but the data acquisition process becomes cumbersome and costly
Solution Approach 1:
The patent uses a generator model to create synthetic copies of real-world depth maps from synthetic dense depth maps. These synthetic raw depth maps replicate the characteristics of real sensor measurements without requiring actual physical data collection, thus resolving the contradiction between data realism and acquisition efficiency
Solution Approach 2:
The generator model transforms synthetic dense depth maps into synthetic raw depth maps by applying parameter changes that simulate sensor measurement characteristics including noise patterns, reflection artifacts, and incomplete sampling. This transformation enables realistic training data generation without physical data collection
2Measurement precision
If depth sensors are used to acquire raw depth maps in industrial settings, then direct 3D geometry measurement is obtained, but large errors and information loss occur due to optical reflections and refractions
Solution Approach 1:
The patent applies preliminary action by using a depth completion model to process and correct raw depth maps before they are used for training. This preliminary processing step compensates for sensor errors and recovers lost information, improving both measurement precision and information completeness
Solution Approach 2:
The depth completion model serves as an intermediary between the flawed sensor measurements and the training data requirements. It mediates by transforming erroneous raw depth maps into corrected depth information, eliminating the direct impact of optical reflections and refractions
3Manufacturing precision
If synthetic dense depth maps are generated from CAD data, then ideal geometric information is obtained, but the data lacks the characteristics of real sensor measurements
Solution Approach 1:
The generator model creates copies of synthetic dense depth maps that incorporate real sensor measurement characteristics. By copying and transforming the ideal geometric data through a learned mapping, the system achieves both geometric accuracy and measurement realism simultaneously
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The present invention relates to a computer-implemented method for providing training data for training of a data-driven depth completion model as a machine-learning model, wherein the depth completion model is to be trained to generate dense depth maps from sensor acquired raw depth maps, comprising the steps of: - providing (S1) multiple synthetic dense depth map data items (Dsyn) from CAD data of various synthetic scenes; - providing (S2) multiple real raw depth map data items (Draw_real) obtained from real-world depth sensor measurements of real-world scenes; - training (S3) a generative model (10) for obtaining a trained generator model (11) for generating generated raw depth map data items (Draw_syn) from the synthetic dense depth map data items (Dsyn); - applying (S4) the trained generator model (11) to generate training data from provided synthetic dense depth map data (Dsyn).