Dense Depth Map Generation From LiDAR and Monocular RGB Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in generating accurate and comprehensive depth maps for human-robot interaction, 3D reconstruction, object detection, and robot navigation, as they often rely on monocular cameras that lack depth information and LiDAR sensors that provide sparse depth maps, which are insufficient for detailed environmental understanding.
Innovation Solution
A method and apparatus that utilize a monocular camera and LiDAR sensor to generate sparse depth maps, which are then input into a pre-trained diffusion model to create dense depth maps, with customizable training settings and post-processing techniques to enhance prediction quality and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If LiDAR sensor is used to acquire depth information, then depth data is obtained, but the depth map remains sparse and insufficient for detailed environmental understanding
Solution Approach 1:
The patent combines LiDAR depth data with RGB image data by projecting 3D point cloud coordinates to 2D image coordinates and merging them into a unified feature space. This fusion allows the sparse LiDAR depth information to be complemented by the dense RGB image data, producing a complete dense depth map that retains accurate depth values where available and infers plausible depth values in sparse regions.
Solution Approach 2:
The patent introduces a diffusion model as an intermediary that processes the merged LiDAR-RGB features. This diffusion model acts as a mediator that transforms the combined sparse depth and dense image features into a dense depth map, filling in missing depth information while preserving the accuracy of existing LiDAR measurements through probabilistic inference.
2Quantity of substance
If monocular camera is used to capture RGB images, then color image data is obtained, but depth information is completely absent
Solution Approach 1:
The diffusion model serves as an intermediary that takes the RGB image data and sparse LiDAR depth information as inputs, and generates dense depth maps by learning the relationship between image appearance and depth structure. The model infers depth values for pixels without direct LiDAR measurements by leveraging patterns learned from training data.
Solution Approach 2:
The patent transforms the RGB image data and sparse depth values into a unified feature representation space through linear projections. This parameter transformation allows the network to process heterogeneous data types (color images and depth values) together and generate dense depth predictions by changing the representation parameters of the input data.
3Ease of manufacture
If sparse depth map is used as training data for diffusion model, then training is simplified, but prediction quality may be compromised without proper normalization
Solution Approach 1:
The patent applies min-max normalization to transform sparse depth values into a standardized range [0, 1] before feeding them to the diffusion model. This parameter transformation ensures that depth values are scaled appropriately, preventing numerical instability during training while preserving the relative depth relationships. The normalized values are then used to train the model, which learns to map these normalized sparse inputs to normalized dense depth outputs.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
The approach enables the generation of dense depth maps with improved prediction quality and accuracy, suitable for various applications, including human-robot interaction, object detection, and robot navigation, by leveraging a diffusion model trained on normalized and conditioned sparse depth maps and filling gaps in depth values.
Implementation Method 1
acquiring a 3D point cloud through a light detection and ranging (LiDAR) sensor provided in a robot system
Data Source
AI summary
A depth map generating method and a depth map generating apparatus are provided. The depth map generating method includes acquiring an RGB color image through a monocular camera provided in a robot system; acquiring a 3D point cloud through a light detection and ranging (LiDAR) sensor provided in the robot system; generating a sparse depth map including only depth information for some points in a given space from the 3D point cloud; inputting the RGB color image and the sparse depth map into a pre-trained diffusion model; and generating a dense depth map including depth information for all points in the given space.