Depth Densification Domain Adaptation for Sparse LiDAR Gaps
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing depth sensors, such as LiDAR, provide sparse and noisy depth data, leading to significant gaps in depth measurements, especially for objects further away, which complicates applications requiring full resolution depth maps.
Innovation Solution
A domain adaptation pipeline trained on synthetic data using geometric sensor imitation, data-specific sensor mimicking, virtual projections, and semi-supervised consistency to enhance depth estimation, allowing the model to learn from sparse to dense depth maps without real-world ground truth.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If sparse depth sensors (LiDAR) are used to determine distance of objects, then measurement range and accuracy for distant objects is improved, but data sparsity and gaps in depth measurements increase
Solution Approach 1:
The patent creates synthetic depth data copies from RGB images using neural network-generated depth maps. These synthetic depth maps serve as virtual copies of what a dense depth sensor would produce, allowing the system to train without requiring actual dense ground truth depth data from expensive sensors.
Solution Approach 2:
The patent replaces the need for expensive physical dense depth sensors with a computational approach using neural networks. Instead of relying on additional hardware sensors to provide dense depth measurements, the system uses learned models to generate synthetic depth maps from standard RGB images.
2Measurement precision
If additional sensors (second camera, IMU, video frames) are used for self-supervised training, then depth estimation accuracy is improved, but system complexity and cost increase
Solution Approach 1:
The patent creates synthetic depth data copies from RGB images using neural network-generated depth maps. These synthetic depth maps serve as virtual copies of what a dense depth sensor would produce, allowing the system to train without requiring actual dense ground truth depth data from expensive sensors.
Solution Approach 2:
The system uses standard RGB cameras to generate their own training data by creating synthetic depth maps through neural networks. The RGB images serve dual purposes: as input for depth estimation and as the basis for generating synthetic ground truth depth maps, making the system self-sufficient without additional sensors.
3Measurement precision
If real ground truth depth data is used for supervised training, then training accuracy is improved, but data acquisition cost and complexity increase
Solution Approach 1:
The patent creates synthetic depth data copies from RGB images using neural network-generated depth maps. These synthetic depth maps serve as virtual copies of what a dense depth sensor would produce, allowing the system to train without requiring actual dense ground truth depth data from expensive sensors.
Solution Approach 2:
The patent uses computationally generated synthetic depth maps as temporary training targets. These synthetic ground truth depth maps are inexpensive to generate compared to collecting real depth data from expensive sensors, and can be created on-demand during the training process without requiring physical data collection campaigns.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
The approach improves depth estimation accuracy by mimicking real-world sensor noise and patterns, effectively filling gaps in sparse depth data, even in the absence of additional sensors, enhancing applications like self-driving vehicles.
Implementation Method 1
Sensing technologies such as RADAR, LiDAR, ultrasound or other time-of-flight techniques utilize the fact that a wave with known properties is emitted and then reflected back from objects with specific density characteristics. If the travelling speed of the wave and the environmental characteristics are known, the echo or the reflection can be used to determine the time the wave took to travel through a medium and then the distance to the points that made the signal reflect can be calculated.
Implementation Method 2
a wave with known properties is emitted and then reflected back from objects with specific density characteristics
Data Source
Figure 1
Figure 2
Figure 3(a)~4(b)
AI summary
A method for training an environmental analysis system, the method comprising: receiving a data model of an environment; forming in dependence on the data model a first training input comprising a visual stream representing the environment as viewed from a plurality of locations; forming in dependence on the data model a second training input comprising a depth stream representing the depth of objects in the environment relative to the plurality of locations; forming a third training input, the third training input being sparser than the second training input; and estimating by means of the analysis system, in dependence on the first and third training inputs, a series of depths at less sparsity then the third training input; and adapting the analysis system in dependence on a comparison between the estimated series of depths and the second training input.