LiDAR-Image Fusion for 3D Target Detection in Autonomous Driving
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing 3D target detection methods in unmanned driving face challenges such as low accuracy in depth measurement, sparse point cloud data, lack of auxiliary information, and slow detection speed, making real-time detection difficult.
Innovation Solution
A method involving joint calibration of LiDAR and image sensors, using two convolutional neural networks to fuse point cloud and image information, extracting multi-feature information, and projecting laser points onto image coordinates for accurate 3D target detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If binocular stereo vision principle is used for depth estimation, then the system complexity is reduced, but the depth measurement accuracy deteriorates significantly
Solution Approach 1:
The patent combines LiDAR point cloud data with image data from cameras to create a fused representation that leverages the depth measurement capabilities of LiDAR and the visual information from images. This merging resolves the contradiction by achieving high depth accuracy through LiDAR while maintaining system complexity management through integrated processing.
Solution Approach 2:
The patent uses an intermediary fusion mechanism that integrates point cloud information with image features. The point cloud serves as an intermediary that bridges the gap between accurate depth measurement and visual recognition, allowing the system to achieve both high measurement precision and manageable complexity.
2Measurement precision
If pure LiDAR point cloud processing is used, then the depth measurement accuracy is improved, but the information richness deteriorates due to sparse point cloud data
Solution Approach 1:
The patent merges LiDAR point cloud data with image data to create a comprehensive representation. This combination resolves the contradiction by supplementing the sparse point cloud with rich visual information from images, including color and texture features, while maintaining the accurate depth measurements provided by LiDAR.
Solution Approach 2:
The patent creates a composite data representation by integrating point cloud information with image features. This composite approach combines the structural accuracy of LiDAR with the information richness of visual data, achieving both high measurement precision and comprehensive information content.
3Loss of information
If LiDAR and image data fusion is performed using existing methods, then the information completeness is improved, but the detection speed deteriorates making real-time detection difficult
Solution Approach 1:
The patent performs preliminary processing of point cloud and image data before fusion, including coordinate transformations and feature extractions. This preliminary action organizes the data in advance, reducing the computational burden during the fusion process and enabling faster real-time detection while maintaining complete information integration.
Solution Approach 2:
The patent segments the fusion process into distinct stages: point cloud processing, image processing, coordinate transformation, and feature fusion. This segmentation allows each stage to be optimized independently, improving overall detection speed while ensuring complete information integration across all stages.
Data Source
Figure 1~2
Figure 3~4
Figure 5~6
AI summary
The present invention provides a detection method, system, and device based on fusion of image and point cloud information, and a storage medium. The method comprises the following steps: synchronously obtaining point cloud information and image information by using a laser radar sensor and an image sensor; inputting the image information into a trained first convolutional neural network to extract multi-feature information of each pixel point in the image information, the multi-feature information at least comprising color information and object identifier information of each pixel point; projecting a laser point cloud of the laser radar sensor onto an image, enabling each laser point in the point cloud information to be matched to a corresponding pixel point, and then adding the multi-feature information of each pixel point into the corresponding point cloud information; and inputting the point cloud information having the multi-feature information into a trained second convolutional neural network to output the category of each 3D target. The present invention can implement real-time 3D target detection having higher precision, and obviously improve the detection precision of a small obj ect.