BEV-Camera Fusion for Autonomous Vehicle Motion Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Autonomous vehicles face challenges in accurately combining data from multiple sensor modalities, leading to decreased accuracy or loss of contextual information necessary for safe and effective operation.
Innovation Solution
A machine learning model is trained to fuse the output of a perception system, such as a synthetic Birds-Eye-View (BEV) image, with data from additional sensor modalities like camera images, using convolutional neural networks to capture contextual information without interfering with the accuracy of the perception system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple sensor modalities are combined to improve contextual information, then motion prediction accuracy is improved, but system complexity increases
Solution Approach 1:
The patent introduces an intermediary machine learning model that acts as a mediator between multiple sensor modalities (LIDAR, camera, radar) and the motion prediction system. This model fuses data from different sensors in a coordinated manner, allowing the system to benefit from multiple sensor inputs without directly managing the complexity of integrating each sensor individually. The intermediary model processes and harmonizes sensor data streams, reducing the computational burden and architectural complexity while maintaining high prediction accuracy.
Solution Approach 2:
The patent segments the complex sensor fusion and motion prediction task into distinct functional modules: a perception system that processes individual sensor inputs, a machine learning model that fuses these perceptions, and a motion prediction system that generates predictions. This segmentation allows each component to be optimized independently and simplifies the overall system architecture by breaking down the complex integration task into manageable, specialized subsystems.
2Productivity
If synthetic BEV images are used to represent environment, then computational efficiency is improved, but loss of contextual information occurs
Solution Approach 1:
The patent merges synthetic BEV images with raw sensor data in the machine learning model. Instead of relying solely on processed BEV images or raw sensor inputs, the system combines both representations, allowing the model to benefit from the computational efficiency of BEV images while simultaneously accessing the rich contextual information present in raw sensor data. This merging occurs in the feature fusion layers of the neural network, where features from both sources are integrated.
Solution Approach 2:
The patent applies local quality by processing different regions of sensor data at different levels of abstraction. The BEV image provides a globally optimized, computationally efficient representation of the environment, while raw sensor data is retained in specific local regions where detailed contextual information is critical for motion prediction. This selective retention of detailed information in specific areas allows the system to maintain computational efficiency overall while preserving necessary contextual details where needed.
Data Source
AI summary
Provided are methods for motion prediction in an autonomous vehicle using fused synthetic and camera images. The method can include obtaining data pairs, each of which reflects data corresponding to a synthetic image representing a birds-eye-view of an area around a vehicle and identifying an object, and data corresponding to a camera image depicting the object. A machine learning model can be trained based on the data pairs to result in a trained model that predicts motion of the object within the data pair based on the synthetic image and camera image in the data pair. Systems and computer program products are also provided.