Depth Map Fusion Using Raw LiDAR and Camera Feature Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for generating depth maps in autonomous vehicles using stereo depth estimation and monocular depth prediction face challenges in accurately measuring depth in meters and efficiently aligning and fusing LIDAR and camera image features with different resolutions.
Innovation Solution
A method that involves extracting LIDAR image features directly from raw LIDAR sensor data, aligning and sampling these features with camera image features using sub-pixel sampling, and fusing them to generate a depth map without requiring separate pre-processing of LIDAR images.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If LIDAR image features and camera image features are aligned and fused to generate depth maps, then measurement precision is improved, but device complexity increases due to the need for separate LIDAR pre-processing and feature alignment mechanisms
Solution Approach 1:
The patent combines LIDAR depth information and camera image features into a unified depth map generation process. The LIDAR point cloud data and camera image features are integrated through feature fusion modules, allowing the system to generate accurate depth maps by merging complementary information from both sensors rather than processing them separately.
Solution Approach 2:
The depth completion network is designed to handle multiple functions: it processes LIDAR point cloud data, integrates camera image features, and generates depth maps for various autonomous driving scenarios. This multi-functional architecture reduces the need for separate specialized processing systems for different sensing modalities.
2Measurement precision
If separate pre-processing of LIDAR images is performed before feature extraction, then feature extraction accuracy is improved, but productivity decreases due to additional processing steps
Solution Approach 1:
The system performs preliminary integration of LIDAR point cloud data with camera image features at the input stage of the depth completion network. By pre-aligning and fusing the sensor data before the main processing pipeline, the network receives integrated features that require less subsequent pre-processing, thereby improving overall processing efficiency while maintaining accuracy.
Solution Approach 2:
The patent extracts only the essential features from LIDAR point cloud data that are most relevant for depth map generation, rather than performing exhaustive pre-processing. The feature extraction module selectively identifies and processes key LIDAR features (such as depth, intensity, and spatial position) that contribute most to depth accuracy, reducing unnecessary computational overhead.
3Productivity
If LIDAR data is directly utilized without separate pre-processing, then productivity is improved, but measurement precision may deteriorate due to potential loss of important LIDAR image features
Solution Approach 1:
The depth completion network acts as an intermediary that receives both LIDAR point cloud data and camera image features, processes them through multiple fusion modules, and generates the final depth map. This intermediary processing structure ensures that important LIDAR features are preserved and properly integrated with camera data, maintaining measurement precision while avoiding the need for separate complex pre-processing pipelines.
Solution Approach 2:
The system dynamically adjusts processing parameters based on the input data characteristics. The network modifies feature extraction parameters, fusion weights, and processing depth according to the quality and characteristics of the LIDAR and camera inputs, allowing it to maintain high accuracy while adapting processing efficiency to the specific scene conditions.
Data Source
AI summary
Disclosed is a processor which includes a camera image feature extractor that extracts a camera image feature based on a camera image, a LIDAR image feature extractor that extracts a LIDAR image feature based on a LIDAR image, a sampling unit that performs a sampling operation based on the camera image feature and the LIDAR image feature and generates a sampled LIDAR image feature, a fusion unit that fuses the camera image feature and the sampled LIDAR image feature and generates a fusion map, and a decoding unit that decodes the fusion map and generates a depth map. The sampling operation includes back-projecting a pixel location of the camera image feature on a camera coordinate system to generate a back-projection point, and projecting the back-projection point on a plane of the LIDAR image to calculate sampling coordinates.