Multi-Sensor Fusion Training for Robust Object Motion Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Autonomous systems face challenges in robust object detection and motion prediction due to increased system complexity and brittleness when using multiple sensor modalities, leading to over-reliance on specific sensors and difficulties in handling sensor noise and simulation data realism gaps.
Innovation Solution
The implementation of a computing system that applies sensor dropout during training to reduce coupling between sensor modalities, allowing for more robust models capable of detecting objects and predicting motion using LIDAR point clouds, camera RGB images, and RADAR returns, while conserving computing resources and improving generalization performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple sensor modalities are used to improve object detection and motion prediction, then sensing performance is improved, but system complexity increases
Solution Approach 1:
The patent divides the sensor processing into separate modalities (LIDAR, camera, RADAR) that are processed independently through dropout training, allowing each sensor type to be optimized separately while maintaining overall system performance
Solution Approach 2:
The patent applies sensor dropout by randomly setting sensor inputs to zero during training with a dropout rate parameter, changing the operational state of sensors dynamically to prevent over-reliance on any single modality
2Measurement precision
If multiple sensor modalities are used to improve sensing capability, then detection accuracy is improved, but model brittleness increases
Solution Approach 1:
The patent prepares the model for sensor failures by applying dropout during training, which cushions the model against brittleness by exposing it to random sensor dropouts before deployment, preventing over-reliance on specific sensors
3Reliability
If sensor dropout is applied during training to reduce coupling between sensor modalities, then model robustness is improved, but training complexity increases
Solution Approach 1:
The patent introduces dynamic randomness into training through dropout, where sensor inputs are randomly zeroed during training but remain available during inference, creating a dynamic training regime that improves robustness without requiring architectural changes
Data Source
AI summary
Systems and methods are disclosed for detecting and predicting the motion of objects within the surrounding environment of a system such as an autonomous vehicle. For example, an autonomous vehicle can obtain sensor data from a plurality of sensors comprising at least two different sensor modalities (e.g., RADAR, LIDAR, camera) and fused together to create a fused sensor sample. The fused sensor sample can then be provided as input to a machine learning model (e.g., a machine learning model for object detection and/or motion prediction). The machine learning model can have been trained by independently applying sensor dropout to the at least two different sensor modalities. Outputs received from the machine learning model in response to receipt of the fused sensor samples are characterized by improved generalization performance over multiple sensor modalities, thus yielding improved performance in detecting objects and predicting their future locations, as well as improved navigation performance.