Vision and Millimeter Wave Fusion for All-Weather Target Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current autonomous driving technologies face challenges in achieving stable environmental perception in complex scenes such as strong light, fog, and night due to the limitations of camera sensors and the intrinsic clutter and data sparsity issues with millimeter wave radars, which affect target detection accuracy.
Innovation Solution
An all-weather target detection method that fuses vision and millimeter wave data by pre-processing continuous frames, combining RCS, distance, and velocity information with visual information using a designed fusion model, and inputting the fused feature map into a YOLOv5 detection network for improved precision and robustness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If camera images are used for environmental perception, then the system can achieve basic target detection functionality, but detection accuracy deteriorates in complex scenes such as strong light, fog, and night
Solution Approach 1:
The patent combines camera images with millimeter wave radar point cloud data to form a fused perception system. The camera provides visual information while the radar provides distance, velocity, and reflection intensity data. By merging these complementary data sources through coordinate transformation and feature fusion, the system overcomes the limitations of camera-only detection in complex environmental conditions.
2Reliability
If millimeter wave radar is added for fusion perception, then reliability in complex scenes is improved, but device complexity increases due to multi-sensor integration
Solution Approach 1:
The patent divides the fusion process into distinct modules: coordinate system calibration, point cloud processing, feature extraction, and detection. By segmenting the complex fusion task into manageable stages, the system reduces implementation complexity while maintaining reliability benefits from multi-sensor integration.
Solution Approach 2:
The patent introduces an intermediate processing stage that transforms radar point cloud data into a format compatible with camera image processing. This includes coordinate transformation to align radar and camera reference frames, and feature extraction that creates a unified representation suitable for joint detection, thereby simplifying the integration of heterogeneous sensors.
3Loss of information
If all millimeter wave data information is combined for fusion perception, then completeness of information is improved, but processing difficulty increases due to intrinsic clutter and data sparsity
Solution Approach 1:
The patent extracts specific useful features from the millimeter wave radar data, including distance, velocity, and reflection intensity, while filtering out clutter and irrelevant information. This selective extraction approach maintains information completeness for detection-relevant parameters while reducing processing complexity by eliminating unnecessary data.
Solution Approach 2:
The patent performs preliminary processing of radar point cloud data before fusion, including coordinate calibration and feature extraction. By preparing the data in advance through these preprocessing steps, the system reduces the complexity of subsequent fusion operations while ensuring all necessary information is properly formatted and aligned.
Data Source
AI summary
An all-weather target detection method based on a vision and millimeter wave fusion includes: simultaneously acquiring continuous image data and point cloud data using two types of sensors of a vehicle-mounted camera and a millimeter wave radar; pre-processing the image data and point cloud data; fusing the pre-processed image data and point cloud data by using a pre-established fusion model, and outputting a fused feature map; and inputting the fused feature map into a YOLOv5 detection network for detection, and outputting a target detection result by non-maximum suppression. The method fully fuses millimeter wave radar echo intensity and distance information with the vehicle-mounted camera images. It analyzes different features of a millimeter wave radar point cloud and fuses the features with image information by using different feature extraction structures and ways, so that the advantages of the two types of sensor data complement each other.