Multisensor Transformer Fusion for 3D Object Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Fusing sensor data from different sensors, such as radar and camera or multiple cameras, results in loss of information due to projection from three-dimensional to two-dimensional space, leading to suboptimal object detection and identification accuracy.
Innovation Solution
A hierarchical arrangement of transformers is employed, where each transformer processes sensor data from a specific modality (e.g., camera or radar) without projecting data between modalities, and a third transformer learns bipartite correlations to fuse features and geometric relations, enhancing object detection and identification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If radar data is projected into two-dimensional space to fuse with image data, then fusion is achieved, but size invariance is lost and information is lost
Solution Approach 1:
The patent projects image data into three-dimensional space to match the dimensional space of radar data, enabling fusion without losing size invariance. This dimensional transformation allows both sensor types to operate in a common 3D coordinate system, preserving geometric relationships while achieving effective fusion.
Solution Approach 2:
The patent introduces a coordinate transformation intermediary that converts image data from 2D to 3D space using camera calibration parameters and depth information. This intermediary transformation enables compatibility between different sensor modalities without direct projection that would cause information loss.
2Reliability
If image data is projected into three-dimensional space to fuse with radar data, then fusion is achieved, but image details are lost and distorted due to depth ambiguity
Solution Approach 1:
The patent performs preliminary feature extraction on image data before projecting it into three-dimensional space. By extracting relevant features (such as object boundaries, textures, and semantic information) prior to the dimensional transformation, the system preserves critical image details that might otherwise be lost or distorted during the projection process.
Solution Approach 2:
The patent utilizes camera calibration parameters and depth estimates to transform image coordinates into 3D space while maintaining geometric accuracy. By carefully managing the transformation parameters and using depth information from radar or stereo vision, the system minimizes distortion and preserves image details during the dimensional conversion.
3Loss of information
If image data from different cameras is projected into a common coordinate system to fuse, then fusion is achieved, but information is lost due to non-overlapping native coordinate systems
Solution Approach 1:
The patent establishes a universal 3D coordinate system that serves as a common reference frame for all camera inputs. By transforming all camera coordinate systems into this universal 3D space using extrinsic calibration parameters, the system enables fusion of data from multiple cameras with different native coordinate systems without losing spatial information.
Solution Approach 2:
The patent creates a 3D representation (copy) of image data from each camera in the universal coordinate system. Instead of trying to align all cameras to a single camera's coordinate system, the system generates 3D point clouds or voxel representations that can be seamlessly integrated into the common 3D space, preserving information from all sensors.
Data Source
AI summary
A system includes a first sensor system of a first modality and a second sensor system of a second modality. The system further includes a computing system that is configured to detect and identify objects represented in sensor signals output by the first and second sensor systems. The computing system employs a hierarchical arrangement of transformers to fuse features of first sensor data output by the first sensor system and second sensor data output by the second sensor system.