Radar-Camera BEV Fusion for Multi-Format 3D Object Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing RADAR systems output different data format types that cannot be radiometrically normalized and aligned in a single model, hindering effective fusion with camera data for 3D object detection.
Innovation Solution
Transform RADAR data into a common birds eye view (BEV) format using an attention mechanism based on camera BEV features, enabling radiometric normalization and alignment of different RADAR data types for fusion with image data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If different RADAR data format types are used, then the system can accommodate diverse RADAR sensor characteristics, but the data cannot be radiometrically normalized and aligned in a single model
Solution Approach 1:
The patent introduces an intermediary normalization layer that transforms diverse RADAR data formats into a unified representation. This intermediary component acts as a mediator between different RADAR data formats and the fusion model, enabling radiometric normalization without requiring separate models for each format type.
Solution Approach 2:
The patent applies parameter changes by dynamically adjusting radiometric parameters (such as intensity scaling and normalization factors) based on the input data format type. This allows the system to adaptively normalize different RADAR formats by modifying key parameters rather than requiring fundamentally different processing pipelines.
2Reliability
If multiple RADAR data format types are fused with camera data, then comprehensive 3D object detection can be achieved, but the fusion process becomes more complex
Solution Approach 1:
The patent segments the fusion process into distinct stages: RADAR data preprocessing and format-specific normalization, camera data processing, and final feature fusion. This segmentation allows each component to be optimized independently while maintaining overall system coherence and reducing the complexity of the integration process.
Solution Approach 2:
The patent implements a universal fusion framework that can handle multiple RADAR data formats through a single multi-functional model. This universal approach enables the system to process diverse RADAR formats (such as range-Doppler maps, range-azimuth maps, and point clouds) alongside camera data using the same underlying architecture.
3Measurement precision
If radiometric normalization is applied to align different RADAR data types, then effective fusion with camera data becomes possible, but the processing time increases
Solution Approach 1:
The patent performs radiometric normalization as a preliminary action during the data preprocessing stage, before the main fusion and detection processes. By pre-normalizing the RADAR data to match the radiometric characteristics of camera data, the system avoids repeated normalization computations during real-time operation, thus reducing processing time while maintaining alignment precision.
Data Source
AI summary
This disclosure provides systems, methods, and devices for vehicle driving assistance systems that support image processing. In a first aspect, a method of image processing includes receiving image BEV features and receiving first radio detection and ranging (RADAR) BEV features. The first RADAR BEV features that are received are determined based on first RADAR data associated with a first data type. First normalized RADAR BEV features are determined, which includes rescaling the first RADAR BEV features using a first attention mechanism based on the image BEV features and the first RADAR BEV features. Fused data is determined that combines the first normalized RADAR BEV features and the image BEV features. Other aspects and features are also claimed and described.