Multi-Modal Sensor Fusion for Cooperative Perception Bandwidth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional cooperative perception systems face challenges in real-time data transmission of raw sensing data between vehicles, leading to decreased detection accuracy due to limitations in detection systems, where objects may not be detected by either vehicle.
Innovation Solution
The system employs intermediate fusion by transmitting features extracted from raw sensing data between vehicles, allowing each vehicle to locally extract features and fuse them without requiring additional computation overhead or raw data transmission, using a scale-fusion network to combine features in a bird's eye view format.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If raw sensing data is transmitted between vehicles for cooperative perception, then detection accuracy can be improved, but data transmission bandwidth requirements increase significantly
Solution Approach 1:
The patent extracts only the essential feature representations from raw sensing data at each vehicle, rather than transmitting the complete raw data. This extraction process isolates the critical information needed for cooperative detection while discarding redundant data, thereby reducing transmission bandwidth requirements while preserving detection accuracy.
Solution Approach 2:
The patent segments the cooperative perception system into independent feature extraction modules at each vehicle, which locally process raw data and transmit only the extracted features. This segmentation allows distributed processing and reduces the need for centralized raw data transmission, achieving both accuracy and bandwidth efficiency.
2Reliability
If prediction results from multiple vehicles are combined, then detection reliability can be improved, but detection accuracy decreases when objects are not detected by either vehicle
Solution Approach 1:
The patent performs preliminary feature extraction and object detection at each vehicle before combining results. By pre-processing the data locally and extracting meaningful features in advance, the system ensures that only relevant detection information is transmitted and combined, improving both reliability and accuracy of the final cooperative detection results.
3Quantity of substance
If feature-based intermediate fusion is used instead of raw data transmission, then data transmission bandwidth is reduced, but computation overhead at each vehicle increases
Solution Approach 1:
The patent changes the parameter representation from raw sensing data to extracted feature representations. This parameter transformation reduces the dimensionality and complexity of transmitted data while maintaining the essential information needed for detection, thereby reducing bandwidth requirements without significantly increasing computation overhead at each vehicle.
Data Source
AI summary
A method of fusing multi-modal sensor data is provided. The method includes obtaining features for 3D data captured by a first sensor of an ego vehicle, obtaining features for images captured by second sensors of the ego vehicle, flattening the features for 3D data to first features in bird eye view, transforming the features for images into second features in bird eye view, concatenating the first features and the second features to obtain first concatenated multi-sensor features, and fusing the first concatenated multi-sensor features with second concatenated multi-sensor features received from another vehicle to obtain fused multi-sensor features.