LiDAR-Camera Feature Fusion for Small-Object Semantic Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
LiDAR-based point cloud semantic segmentation for vehicles faces challenges in object recognition due to increased spacing between beams with distance, reduced point representation for small objects, and lack of color information, with traditional data fusion methods struggling to accurately match LiDAR and camera feature maps.
Innovation Solution
A method and system that applies voxel and pixel matching-based fusion, using a 3D voxel-based neural network to generate voxel-wise feature maps from LiDAR data and a neural network for pixel-wise feature maps from camera data, with calibration parameters to align coordinates and combine features via convolutional layers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Area of stationary object
If LiDAR beam spacing is increased to cover larger area, then coverage area is improved, but measurement precision deteriorates due to reduced point representation for small objects
Solution Approach 1:
The patent merges LiDAR point cloud data with camera image data to create fused feature maps. The camera provides high-resolution color information that complements the geometric data from LiDAR, allowing small objects to be detected with both geometric precision and visual detail, thus maintaining measurement precision while covering large areas.
Solution Approach 2:
The patent transitions from 3D point cloud data to 2D feature maps that can be processed by convolutional neural networks. By projecting 3D LiDAR data onto 2D planes and fusing with 2D camera images, the system maintains detailed object representation while enabling efficient processing of large-scale environmental data.
2Length of stationary object
If LiDAR distance to object increases, then detection range is improved, but measurement precision deteriorates due to greater beam spacing
Solution Approach 1:
The patent combines LiDAR depth information with camera image data to create fused feature maps. Even when LiDAR beam spacing increases at greater distances, the high-resolution camera images provide complementary visual information that maintains measurement precision for distant objects through the fusion process.
3Productivity
If traditional encoder-based feature fusion is used, then processing speed is improved, but measurement precision deteriorates due to inaccurate matching between LiDAR and camera feature maps
Solution Approach 1:
The patent performs preliminary coordinate transformation and feature map generation before fusion. By pre-processing LiDAR point cloud data into structured feature maps and aligning them with camera images using calibration parameters, the system establishes accurate correspondences between sensors before the main fusion operation, ensuring both speed and precision.
Data Source
AI summary
A LiDAR and camera data fusion method includes generating a voxel-wise feature map based on point cloud data of a LiDAR sensor, generating a pixel-wise feature map based on image data of a camera, converting three-dimensional (3D) coordinates of point data of the voxel-wise feature map to two-dimensional (2D) coordinates, based on at least one predefined calibration parameter, and generating fused data by combining pixel data of the pixel-wise feature map and point data of the 2D coordinates.