Camera Radar Sensor Fusion 3D Feature Map Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing autonomous driving technologies face limitations in accurately perceiving a 3-D driving environment, especially in varying weather conditions and with single sensor failures, due to the late fusion method of camera and radar sensor data, which leads to loss of information and reduced performance.
Innovation Solution
A sensor fusion-based 3-D environment perception algorithm that combines camera and radar sensor data to generate a 3-D feature map, overcoming the limitations of single sensors by extracting 2-D feature maps from camera images and transforming them into 3-D space using distance information from both camera images and radar sensors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a late fusion method is used to combine camera and radar sensor data, then the system can independently process each sensor's data, but information is lost during independent detection and performance is reduced
Solution Approach 1:
The patent merges camera and radar sensor data at the feature level rather than performing independent detection followed by late fusion. The camera feature extractor and radar feature extractor operate in parallel but their features are combined early in the network architecture, allowing information from both sensors to be integrated before final object detection, thereby preventing information loss while maintaining independent processing capabilities
Solution Approach 2:
The patent transforms radar data from its native format into a bird's-eye-view (BEV) feature map representation that matches the spatial dimensionality of camera features. This dimensional transformation allows radar information to be seamlessly integrated with camera data in the same feature space, enabling effective fusion without information loss
2Quantity of substance
If camera and radar sensors are used for object detection, then semantic information and distance information can be provided, but the system is limited to specific environments and minimum speeds
Solution Approach 1:
The patent employs parameter changes by dynamically adjusting the fusion strategy based on environmental conditions and sensor performance. The system modifies weighting parameters and fusion coefficients adaptively, allowing it to maintain high detection accuracy across varying environments (e.g., different weather conditions, lighting scenarios, and speeds) while utilizing the complementary information from both camera and radar sensors
3Measurement precision
If sensor fusion is performed to improve 3-D environment perception, then accuracy can be enhanced, but system complexity increases
Solution Approach 1:
The patent segments the sensor fusion process into distinct functional modules: a camera feature extractor that processes image data, a radar feature extractor that processes radar returns, and a fusion module that combines their outputs. This segmentation allows each component to be optimized independently while maintaining overall system accuracy, managing complexity through modular architecture
Solution Approach 2:
The patent introduces an intermediary bird's-eye-view feature map representation that mediates between camera and radar data. This intermediate representation serves as a common language that both sensor types can contribute to, simplifying the fusion process by providing a standardized interface rather than directly combining disparate sensor formats
Data Source
AI summary
A system and method for perceiving a 3-D environment by using a camera and a radar sensor are disclosed. In order to generate a 3-D feature map that is used to perceive an environment through the fusion of a camera and a radar sensor, the method of perceiving a 3-D environment may include extracting a two-dimensional (2-D) feature map from an image obtained by the camera and transforming the 2-D feature map into a feature map in a 3-D space by using first distance information extracted from the image and second distance information measured by the radar sensor.