Vehicle Object Trajectory Prediction With Multimodal Sensor Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for predicting the future position of objects in autonomous vehicles face challenges due to limited viewing angles from egocentric views and the need for extensive data processing, especially when using bird's-eye views or single-modal predictions, which can lead to inefficiencies in accident prevention and response.
Innovation Solution
A method that combines multimodal synthesis using wide-view RGB 2D images and LiDAR information through a deep learning-attention mechanism, generating a Gaussian mixture probability distribution to predict the future position of objects by mixing video, semantic segmentation, and LiDAR data, thereby overcoming the limitations of single-modal predictions and limited viewing angles.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If egocentric view with single-view RGB camera is used, then device complexity is reduced, but measurement precision and viewing coverage deteriorate due to partial and narrow viewing angle
Solution Approach 1:
The patent combines multiple sensing modalities (RGB cameras for visual information and LiDAR for depth information) into a unified prediction system. The RGB camera captures wide-view images while LiDAR provides accurate depth data, and their features are fused through a deep learning model to predict future object positions, thereby overcoming the limitations of single-modal predictions
Solution Approach 2:
The patent transitions from 2D image data to 3D spatial understanding by integrating LiDAR depth information with RGB image data. This dimensional enhancement allows the system to accurately predict future positions of objects in 3D space, resolving the position ambiguity inherent in single-view 2D images
2Measurement precision
If bird's-eye view or multi-modal data processing is used, then object position prediction accuracy is improved, but data processing complexity and computational load increase
Solution Approach 1:
The patent performs preliminary processing of RGB and LiDAR data to extract relevant features before fusion. By pre-processing inputs to identify key object features and characteristics, the system reduces the complexity of subsequent multi-modal fusion operations while maintaining high prediction accuracy
Solution Approach 2:
The patent introduces a deep learning-based fusion model as an intermediary that systematically integrates features from multiple modalities. This intermediary component manages the complexity of multi-modal data processing by providing a structured approach to feature fusion and prediction
3Measurement precision
If extensive multi-modal data processing is performed, then prediction accuracy is improved, but response time and computational efficiency deteriorate
Solution Approach 1:
The patent extracts only the essential features from RGB and LiDAR data that are most relevant for predicting object future positions. By selectively extracting key features rather than processing all available data, the system maintains high prediction accuracy while reducing computational time and resource requirements
Solution Approach 2:
The patent transforms multi-modal sensor data into standardized feature representations that are optimized for efficient processing. By changing the parameter representation of input data into suitable feature formats, the system accelerates computation while preserving predictive information
4Productivity
If single-modal prediction methods are used, then computational efficiency is improved, but prediction reliability and accuracy deteriorate
Solution Approach 1:
The patent merges predictions from multiple modalities (RGB visual features and LiDAR depth features) into a unified prediction result. This combination leverages the complementary strengths of different sensing modalities, improving prediction reliability while maintaining computational efficiency through integrated processing
Data Source
AI summary
In a method of predicting a position of an object at a future time point for a vehicle, video image information at a current time point and at a plurality of time points before the current time point acquired through a camera of the vehicle may be extracted as semantic segmentation image. A mask image imaging an attribute and position information of an object present in each of the video images may be extracted. A position distribution of the object may be predicted by deriving a plurality of hypotheses for a position of the object at a future time point through deep learning by receiving video images at the current time point and the time points before the current time point, a plurality of semantic segmentation images, a plurality of mask images, and ego-motion information of the vehicle, and calculating the plurality of hypotheses as a Gaussian mixture probability distribution.