Lidar-Camera Data Fusion Using Angle-Triggered Image Capture
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data fusion methods for unmanned devices, such as self-driving vehicles, fail to effectively combine the accurate depth information from lidar point clouds with the abundant color information from image sensors, leading to incomplete data for precise control and navigation.
Innovation Solution
A data fusion method and system that utilizes a processor to synchronize the acquisition of images from image sensors with point clouds from a rotating lidar by sending trigger signals based on predetermined rotation angle intervals, and fuses the pixel information from images with point cloud data using pose change information to align and combine the data accurately.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If image data is used, then color information is abundant, but depth information is inaccurate
Solution Approach 1:
The patent merges image data from image sensors with point cloud data from lidar sensors to create fused data that combines both color information and depth information. The fusion process aligns pixels from images with corresponding points from point clouds based on camera intrinsics and spatial transformation matrices, resulting in data that has both abundant color information and accurate depth information.
2Measurement precision
If point cloud data is used, then depth information is accurate, but color information is insufficient
Solution Approach 1:
The patent combines point cloud data with image data by matching points from the point cloud with corresponding pixels from images using camera intrinsics and spatial transformation. This merging process transfers color information from images to the structured point cloud data, resulting in enriched data that has both accurate depth information and abundant color information.
3Loss of information
If data from multiple sensors is fused, then information completeness is improved, but system complexity increases
Solution Approach 1:
The patent performs preliminary calibration of camera intrinsics and establishes spatial transformation relationships between sensors before actual data fusion. These pre-computed parameters (camera intrinsics matrix, extrinsic transformation matrix) are stored and reused during operation, simplifying the real-time fusion process while ensuring accurate alignment of multi-sensor data.
Solution Approach 2:
The patent uses camera intrinsics and spatial transformation matrices as intermediary components that mediate between different sensor coordinate systems. These intermediaries translate and align data from image sensors and lidar sensors into a common reference frame, enabling accurate fusion while managing system complexity through standardized transformation operations.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
The method provides fused data with both accurate distance and color information, enhancing the precision of unmanned device control, obstacle detection, and trajectory planning, as well as improving the accuracy of high-precision mapping and machine learning model training.
Implementation Method 1
a lidar and at least one image sensor that are arranged on the vehicle, the lidar is configured to acquire a point cloud by rotating a laser emitter
Implementation Method 2
obtaining a rotation angle of the laser emitter determined by a rotation angle measurer of the lidar
Data Source
AI summary
A data fusion method is provided. In one embodiment, the method comprises: acquiring a rotation angle of a laser transmitter of a lidar; selecting, according to a predetermined correspondence between rotation angle intervals and image sensors, an image sensor corresponding to a rotation angle interval in which the obtained rotation angle of the laser emitter is located as a specified image sensor, sending a trigger signal to the specified image sensor, to enable the specified image sensor to acquire an image, receiving the image and a point cloud that is acquired and returned by the lidar within the rotation angle interval in which the obtained rotation angle is located, and fuse information of pixels in the image and information of points in the point cloud according to pose change information of a vehicle in a process of acquiring the image and the point cloud.