Semantic-Point Grid Fusion for Radar-Led Object Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing self-driving systems face challenges in effectively fusing lidar and camera data, particularly in occluded and adverse weather conditions, with radar-based systems struggling to recognize objects due to lack of rich texture and semantic features, and current fusion methods failing to leverage radar data fully.
Innovation Solution
A system that performs sequential fusion by decoupling feature extraction from camera and depth data sensors, using a Semantic-Point-Grid (SPG) encoding to combine radar point clouds with camera-derived semantic information, ensuring reliable object detection even in adverse conditions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If radar data is projected onto camera perspective view RGB image data, then fusion between radar and camera is achieved, but it becomes harder to distinguish between small objects close to the sensor and large objects at a longer range
Solution Approach 1:
The patent transforms radar data from perspective view to Bird's Eye View (BEV) representation, changing the dimensional perspective from camera-like 2D projection to a top-down 3D俯瞰 view. This dimensional transformation allows objects at different distances to be represented with consistent spatial scaling, eliminating the size-distance confusion inherent in perspective views where distant large objects and close small objects appear similar in size.
2Reliability
If inverse mapping of camera's RGB image to radar's Bird's eye view is performed, then fusion is achieved, but the mapping is ill-defined due to lack of depth information in camera images
Solution Approach 1:
Instead of attempting to map camera images to BEV (which is ill-defined due to missing depth), the patent inverts the approach by projecting radar BEV data onto camera perspective views. This reversal leverages the strength of radar in providing accurate depth and distance information while using camera images only for semantic enrichment, making the mapping process well-defined and robust.
3Reliability
If simultaneous feature extraction from camera and radar is performed, then feature-level fusion is achieved, but the system has heavy reliance on cameras and performance severely degrades when camera input is degraded
Solution Approach 1:
The patent introduces BEV representation as an intermediary framework that decouples the fusion process from camera dependency. By performing feature extraction and fusion in the BEV space rather than simultaneously in camera space, the system allows radar to serve as the primary sensor while camera features act as supplementary information. This intermediary representation enables the system to maintain robust performance in adverse conditions where camera input may be degraded.
Data Source
AI summary
A system for image fusion with a depth data includes an imaging system that provides image data with semantic information. A depth data sensor system provides depth data of objects in a field of view. A processor independently extracts the semantic information from the imaging system and combines it with the depth data by assigning weights. The processor generating a semantic-point encoding with depth data as central data. The central data can then play the primary role in object identification, while the system retains depth data and image data for use when the other is insufficient in view of the conditions during sensing. The depth data preferably is point cloud data, such as data from a mechanical radar that is processed to provide point cloud data or a radar system that provides point cloud data.