Sensor Fusion Neural Network for Autonomous Object Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current object detection methods in autonomously driving vehicles lack efficiency and reliability, particularly in fusing data from multiple sensors like lidar, radar, and cameras for accurate 3D and 2D bounding box detection.
Innovation Solution
A computer-implemented method that acquires and processes lidar, radar, and camera data by aligning and concatenating them using ResNet-type neural networks, with residual blocks and linear depth completion, to enhance sensor data fusion for improved object detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple sensor data sets (lidar, radar, camera) are fused for object detection, then detection accuracy and reliability are improved, but system complexity and computational requirements increase
Solution Approach 1:
The system divides the complex sensor fusion task into separate processing streams for camera data, lidar data, and radar data. Each sensor type is processed independently through its own neural network backbone, and features are concatenated at the feature level rather than fusing raw sensor data. This segmentation reduces computational complexity while maintaining detection reliability.
Solution Approach 2:
The patent introduces an intermediary feature concatenation layer that combines features from separate camera, lidar, and radar processing streams. Instead of directly fusing complex sensor data, the system uses intermediate feature representations as mediators, which simplifies the integration process and reduces computational burden while preserving the benefits of multi-sensor fusion.
2Loss of information
If multiple previous sensor sweeps are aligned and concatenated into single frames, then sensor data density and information content are enhanced, but processing time and computational load increase
Solution Approach 1:
The system performs preliminary alignment and concatenation of multiple sensor sweeps into enriched single frames before the main detection process. By pre-processing and combining historical sensor data into consolidated frames with enhanced information content, the system reduces the computational load during real-time detection while maintaining access to dense temporal information.
Solution Approach 2:
The patent applies partial action by selectively processing and concatenating a limited number of previous sensor sweeps (e.g., 2-3 previous frames) rather than all available historical data. This provides sufficient information enhancement for accurate detection while avoiding excessive processing time and computational resources.
3Productivity
If lidar and radar data are cast and projected onto camera images, then data fusion is simplified and processing is accelerated, but measurement precision may be reduced
Solution Approach 1:
The patent casts and projects 3D lidar point cloud data and radar data onto the 2D camera image plane, transforming data from three-dimensional space to two-dimensional space. This dimensionality change simplifies the fusion process by providing a common reference frame, accelerates processing through 2D convolutional operations, and maintains sufficient precision for detection tasks by preserving spatial relationships in the projected domain.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A computer implemented method for object detection comprises the following steps carried out by computer hardware components: acquiring a plurality of lidar data sets from a lidar sensor; acquiring a plurality of radar data sets from a radar sensor; acquiring at least one image from a camera; determining concatenated data based on casting the plurality of lidar data sets and the plurality of radar data sets to the at least one image; and detecting an object based on the concatenated data.