Autonomous Vehicle Motion Prediction With Fused BEV and Camera Images
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Autonomous vehicles face challenges in accurately combining data from multiple sensor modalities, leading to a trade-off between decreased accuracy and loss of contextual information, particularly when using lidar for producing birds-eye-view synthetic images.
Innovation Solution
A machine learning model is trained to fuse synthetic BEV images with data from additional sensor modalities like camera images, using convolutional neural networks to enhance motion prediction and planning by incorporating contextual information without interfering with the perception system's accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If synthetic BEV images from lidar are used for motion prediction, then spatial awareness and object detection are improved, but contextual information from other sensor modalities is lost
Solution Approach 1:
The patent combines synthetic BEV images from lidar with images from additional sensor modalities (cameras, radar) to create a fused representation that preserves both spatial awareness and contextual information. The machine learning model processes multiple input types simultaneously, merging their complementary strengths rather than relying on a single sensor modality.
Solution Approach 2:
The machine learning model is designed to accept and process multiple types of input data from different sensor modalities, making it universally applicable to various sensor types. This multi-functional approach allows the system to leverage diverse sensor information for comprehensive motion prediction and scene understanding.
2Loss of information
If multiple sensor modalities are combined for motion prediction, then contextual information is improved, but accuracy of individual perception systems decreases
Solution Approach 1:
The patent uses synthetic BEV images as an intermediary representation that bridges different sensor modalities. These synthetic images serve as a common language that allows information from cameras, radar, and lidar to be integrated without directly interfering with the accuracy of individual perception systems. The machine learning model processes this intermediary representation along with other sensor data.
3Adaptability or versatility
If additional sensor modalities are integrated into the perception system, then motion prediction capability is improved, but system complexity increases
Solution Approach 1:
The patent segments the complex task of multi-sensor integration into distinct processing streams. The system separately processes synthetic BEV images, camera images, and other sensor data through dedicated neural network branches before fusing the results. This segmentation manages complexity by organizing processing tasks into modular, manageable components.
Data Source
AI summary
Provided are methods for motion prediction in an autonomous vehicle using fused synthetic and camera images. The method can include obtaining data pairs, each of which reflects data corresponding to a synthetic image representing a birds-eye-view of an area around a vehicle and identifying an object, and data corresponding to a camera image depicting the object. A machine learning model can be trained based on the data pairs to result in a trained model that predicts motion of the object within the data pair based on the synthetic image and camera image in the data pair. Systems and computer program products are also provided.