Transformer-Based Object Tracking for Multi-Sensor Detection Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Autonomous vehicles face challenges in accurately combining and processing sensor data from different modalities due to discrepancies and errors, leading to inefficient object detection and tracking, which can impede safe navigation.
Innovation Solution
Utilizing a machine-learned transformer model with self-attention to determine associations and refine object detections from multiple sensor modalities, eliminating the need for pre-processing and post-processing steps typically required by CNN-based systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If sensor data from multiple modalities is combined to improve object detection accuracy, then detection reliability is improved, but computational complexity and processing time increase
Solution Approach 1:
The system segments sensor data processing by creating separate object detection pipelines for different sensor modalities (lidar, radar, vision) that operate independently to generate initial detections, then uses a transformer model to associate detections across modalities. This segmentation allows each modality to be processed optimally without requiring complex real-time fusion of raw sensor data, improving detection reliability while managing computational complexity.
Solution Approach 2:
The patent introduces an intermediary association mechanism that matches object detections from different sensor modalities based on spatial proximity and temporal consistency. This intermediary layer reconciles discrepancies between modalities by identifying corresponding objects across sensors without requiring direct real-time fusion of all sensor data streams, thereby improving detection reliability while avoiding the computational burden of processing all modalities simultaneously.
2Measurement precision
If sensor data from multiple modalities is combined to reduce measurement errors, then object tracking precision is improved, but processing time increases
Solution Approach 1:
The system performs preliminary object detection independently for each sensor modality before association. Each modality generates its own set of object detections with associated attributes (position, velocity, size) in advance, allowing the association stage to focus only on matching pre-processed detections rather than processing raw sensor data from multiple modalities simultaneously. This preliminary action reduces processing time while maintaining tracking precision.
Solution Approach 2:
The transformer model changes the parameter representation of object detections by encoding spatial coordinates, temporal information, and detection attributes into a unified feature space. This parameter transformation enables efficient comparison and association of detections across different modalities by focusing on key discriminative features rather than raw sensor measurements, thereby improving tracking precision without proportionally increasing processing time.
3Reliability
If pre-processing and post-processing steps are added to combine sensor data, then detection accuracy is improved, but device complexity increases
Solution Approach 1:
The patent merges the association of multi-modal object detections with the existing object tracking pipeline into a unified transformer model. Rather than implementing separate pre-processing, association, and post-processing modules, the system integrates these functions into a single end-to-end model that processes detections from multiple modalities and outputs associated object tracks. This merging reduces system complexity by eliminating multiple processing stages while maintaining detection accuracy.
4Reliability
If multiple sensor modalities are used to navigate safely in congested areas, then navigation safety is improved, but computational overhead increases
Solution Approach 1:
The system segments the computational workload by having each sensor modality independently generate object detections before association. This segmentation allows parallel processing of sensor data streams and enables the use of computationally efficient, modality-specific detection algorithms that can operate independently, reducing peak computational overhead while maintaining the safety benefits of multi-modal sensing for navigation decisions.
Data Source
AI summary
Object detection and tracking systems may use machine-learned transformer models with self-attention for detecting, classifying, and/or tracking objects in an environment. Techniques described herein may include receiving sensor data generated by different sensor modalities of a vehicle, determining different bounding shapes based on the different sensor modalities, and using a machine-learned transformer model to determine associated and/or combined bounding shapes. The machine-learned transformer model may receive a variable number of input bounding shapes representing any number of objects and various sensor modalities. Multiple stages of the transformer may be used to determine associated bounding shapes and to assign attributes for the associated bounding shapes, based on the individual bounding shapes of the different sensor modalities and/or previous bounding shapes for objects detected and tracked in a previous scene in the environment.