Early Camera-Radar Fusion in a Common Spatial Domain
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current radar and camera sensor systems in vehicles have limitations in object detection, with radar systems lacking resolution to identify object features and camera systems requiring additional processing stages for accurate classification, necessitating a method to effectively fuse data for enhanced performance in vehicle safety applications.
Innovation Solution
The method involves receiving camera and radar frames, performing feature extraction processes to generate feature maps, converting them to a common spatial domain, concatenating the maps, and using an encoder-decoder network to combine the data for object detection, enabling improved object recognition and classification in spatial domains.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If radar systems are used to detect objects, then measurement of object distance and velocity is accurate, but resolution to identify object features is insufficient
Solution Approach 1:
The patent merges radar and camera data at the feature map level by concatenating radar feature maps with camera feature maps in a common spatial domain. This combination allows the system to retain radar's accurate distance and velocity measurements while supplementing them with camera's rich visual features for object identification, thereby resolving the contradiction between measurement precision and information completeness.
2Loss of information
If camera sensors are used to identify object features, then resolution to identify object features is sufficient, but additional processing stages are required for accurate classification
Solution Approach 1:
The patent performs preliminary feature extraction on camera frames to generate camera feature maps before fusion. By pre-processing the camera data into meaningful feature representations and aligning them with radar feature maps in advance, the system reduces the complexity of subsequent classification stages while maintaining sufficient object feature identification capability.
3Ease of operation
If radar and camera data are fused at high level, then each source yields a decision, but the fusion is less informative than early fusion
Solution Approach 1:
The patent segments the fusion process into distinct levels: early fusion at the feature map level and later fusion at the decision level. By segmenting the fusion operations, the system can perform early fusion to create informative combined feature representations while maintaining the ability to perform decision-level fusion for final classification, thus preserving both information content and operational ease.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach enhances object detection accuracy and reduces computational costs by integrating radar and camera data at an early fusion stage, providing a more informative output for vehicle safety systems like adaptive cruise control and collision avoidance.
Implementation Method 1
Radar systems utilize radio waves to determine the range, altitude, direction, and/or speed of the objects along the road. A transmitter transmits pulses of radio waves that bounce off of objects in their path. The pulses reflected from the objects return a small part of the radio wave's energy to a receiver
Data Source
AI summary
Disclosed are techniques for fusing camera and radar frames to perform object detection in one or more spatial domains. In an aspect, an on-board computer of a host vehicle receives, from a camera sensor of the host vehicle, a plurality of camera frames, receives, from a radar sensor of the host vehicle, a plurality of radar frames, performs a camera feature extraction process on a first camera frame of the plurality of camera frames to generate a first camera feature map, performs a radar feature extraction process on a first radar frame of the plurality of radar frames to generate a first radar feature map, converts the first camera feature map and/or the first radar feature map to a common spatial domain, and concatenates the first radar feature map and the first camera feature map to generate a first concatenated feature map in the common spatial domain.