BEV Sensor Fusion for Robust Scenario Sensing Under Data Loss
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing autonomous driving systems face inaccuracies in scenario sensing due to strong coupling between image data and point cloud data, leading to inaccurate sensing results when one or both data types are missing or of poor quality.
Innovation Solution
A scenario sensing method utilizing a target model that performs feature extraction and fusion using BEV features, incorporating a deformable attention mechanism and mixture of experts architecture to handle missing or low-quality data from cameras and lidars, ensuring accurate sensing results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If image data and point cloud data are strongly coupled in the neural network model, then the sensing result accuracy is improved when both data types are available, but the sensing result becomes inaccurate when one or both data types are missing
Solution Approach 1:
The patent segments the feature processing into independent image feature extraction and point cloud feature extraction pathways. Each pathway processes its respective data type independently through separate neural network branches, allowing one pathway to compensate when the other is unavailable. The segmented features are then fused at a later stage, maintaining independence while enabling integration when both data types are present.
Solution Approach 2:
The patent introduces a feature fusion module as an intermediary between the independent image and point cloud processing pathways. This fusion module acts as a mediator that can adaptively combine features from both data types when available, or rely on features from a single data type when the other is missing, thereby maintaining sensing accuracy under varying data availability conditions.
2Measurement precision
If the neural network model requires both image data and point cloud data for feature fusion, then the sensing precision is improved, but the system complexity increases
Solution Approach 1:
The patent implements a dynamic feature fusion mechanism that adapts to the availability of input data. The model structure dynamically adjusts its processing pathway: when both image and point cloud data are available, it performs full dual-branch feature fusion; when one data type is missing, it automatically switches to single-branch processing. This dynamic adaptation maintains high sensing precision while avoiding the need for a permanently complex dual-branch structure.
3Measurement precision
If the system uses advanced feature processing techniques like deformable attention mechanism and mixture of experts architecture, then the sensing accuracy is maintained under data loss, but the computational complexity increases
Solution Approach 1:
The patent employs parameter changes in the form of learnable attention weights and expert selection probabilities. The deformable attention mechanism dynamically adjusts attention parameters based on input data characteristics, while the mixture of experts architecture selectively activates different expert networks based on the situation. These parameter changes enable the model to maintain high accuracy by adapting to data quality and availability without requiring all computational resources to be constantly engaged.
Data Source
AI summary
This application discloses a scenario sensing method and a related device. The method includes: when scenario sensing needs to be performed on the target scenario, image data of the target scenario and/or point cloud data of the target scenario may be obtained, and the image data and/or the point cloud data may be input into a target model; the target model may first perform feature extraction on the image data and/or the point cloud data to obtain a first BEV feature of the image data and/or a second BEV feature of the point cloud data; the target model may perform feature fusion on the first BEV feature and/or the second BEV feature based on a preset third BEV feature to obtain a feature fusion result; and finally, the target model may perform further processing on the feature fusion result to obtain a sensing result of the target scenario.