Sensor Fusion CNN for Autonomous Driving Object Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current autonomous driving technologies face limitations with individual sensors like Lidar, radar, and camera, with Lidar being expensive, radar having low performance when used alone, and cameras being unstable due to environmental factors, necessitating a method for substantial sensor fusion.
Innovation Solution
A learning method and device that integrate information from a radar capable of distance estimation and a camera using a Convolutional Neural Network (CNN) through channel-wise concatenation of radar and camera data to enhance neural networks for autonomous driving, generating a multichannel image for improved object detection and segmentation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If sensor fusion is implemented to improve detection reliability, then object detection reliability is improved, but device complexity increases
Solution Approach 1:
The patent merges radar and camera data processing into a unified CNN architecture. The radar point cloud data and camera image data are concatenated channel-wise to form a combined input tensor, allowing the neural network to process both sensor types simultaneously through shared convolutional layers. This merging approach improves detection reliability by combining complementary sensor information while managing complexity through architectural integration rather than separate processing pipelines.
Solution Approach 2:
The CNN architecture is designed to handle multiple sensor types universally. The same convolutional layers process both radar-derived feature maps and camera-derived feature maps, with the network learning to extract relevant features from each sensor type through shared weights. This multi-functional design allows a single model to leverage information from both radar and camera, improving reliability without requiring separate specialized processors for each sensor.
2Loss of information
If radar data is integrated with camera data through channel-wise concatenation, then information completeness is improved, but data processing complexity increases
Solution Approach 1:
The patent transforms radar point cloud data into a 2D feature map representation that matches the spatial dimensions of camera images. By projecting 3D radar data onto a 2D grid and concatenating it channel-wise with the 2D camera image, the system combines information from different dimensional representations (3D spatial vs. 2D spatial) into a unified 2D tensor format. This dimensional transformation enables comprehensive information integration while maintaining compatibility with standard 2D CNN architectures, managing processing complexity through dimensional unification.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method for training a CNN by using a camera and a radar together, to thereby allow the CNN to perform properly even when an object depiction ratio of a photographed image acquired through the camera is low due to a bad condition of a photographing circumstance is provided. And the method includes steps of: (a) a learning device instructing a convolutional layer to apply a convolutional operation to a multichannel integrated image, to thereby generate a feature map; (b) the learning device instructing an output layer to apply an output operation to the feature map, to thereby generate estimated object information; and (c) the learning device instructing a loss layer to generate a loss by using the estimated object information and GT object information corresponding thereto, and to perform backpropagation by using the loss, to thereby learn at least part of parameters in the CNN.