Sensor-Agnostic Depth Map Inference With Sparse Prompting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for predicting depth maps from single images often yield inaccurate results when the image deviates from the training dataset or camera parameters, and methods using active sensors like LiDAR and ToF provide sparse depth maps with limited depth values.
Innovation Solution
A model learning method and system that uses a depth map inference model comprising a base model and a prompt model to generate a sparse depth map from a ground truth depth map, training the model to overcome biases and predict a dense depth map independent of sensor type, by fusing image features with sparse depth map features through prompt encoding.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional depth map prediction methods are used for images deviating from training data distribution, then model generalization is required, but prediction accuracy deteriorates
Solution Approach 1:
The method performs preliminary actions by generating a sparse depth map from ground truth depth values before final depth map prediction. This sparse depth map serves as a preliminary structural framework that guides the subsequent dense depth map generation, ensuring both generalization capability and prediction accuracy when images deviate from training data distribution
Solution Approach 2:
The sparse depth map acts as an intermediary between the input image and the final dense depth map prediction. By introducing this intermediate representation that captures key depth structures, the method bridges the gap between generalization requirements and accuracy demands, allowing the model to adapt to new images while maintaining precise depth predictions
2Speed
If active sensors like LiDAR and ToF are used to capture depth maps, then real-time depth acquisition is achieved, but depth map density deteriorates to sparse values
Solution Approach 1:
The method merges the sparse depth map captured by active sensors with the dense depth map predicted by the neural network model. By combining these two depth maps through feature fusion and collaborative prediction, the system achieves both real-time acquisition speed and dense depth map output, resolving the contradiction between speed and density
Solution Approach 2:
The final depth map is constructed as a composite of features from both the sparse sensor data and the dense predicted data. This composite approach integrates the real-time measurement advantages of active sensors with the completeness of dense depth maps, achieving both high speed and high density in the output
3Measurement precision
If sensor-specific depth maps are used for training, then training data accuracy is improved, but sensor type adaptability deteriorates
Solution Approach 1:
The method employs a universal depth map inference model that can process and adapt to depth maps from different sensor types (LiDAR, ToF, structured light, etc.). The model is designed with multi-functionality to handle various sensor input formats and characteristics, enabling it to maintain high adaptability while achieving accurate predictions across different sensor platforms through sensor-agnostic feature extraction and fusion mechanisms
Data Source
AI summary
A model learning method capable of sensor-agnostic depth map inference is provided. The model learning method includes receiving a training image and a ground truth depth map, generating a sparse depth map for training corresponding to the ground truth depth map, generating, using a first model provided to predict a depth map, a first feature and a first depth map corresponding to the training image, substituting the first depth map, which is a relative depth map acquired from the first model, with an absolute depth map reflecting the sparse depth map for training, generating, using a second model provided to perform prompt encoding, a second depth map corresponding to the sparse depth map for training, the first feature, and the first depth map, and training the second model so that the second depth map simulates the ground truth depth map.