Scenario Sensing BEV Fusion for Missing Sensor Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing autonomous driving systems face inaccuracies in scenario sensing due to strong coupling between image and point cloud data, leading to inaccurate results when one of the data types is missing or of poor quality.
Innovation Solution
A scenario sensing method using a target model that performs feature extraction and fusion of BEV features from image and point cloud data, employing deformable attention mechanisms and mixture of experts architectures to ensure accurate sensing results even with incomplete or low-quality data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep learning models with large number of parameters are deployed on mobile devices, then model accuracy is improved, but model size and computational complexity increase
Solution Approach 1:
The patent segments the model deployment process into multiple stages: training phase on server, distillation phase where knowledge is transferred from large teacher model to smaller student model, and deployment phase on mobile device. This segmentation allows the final deployed model to be small while still capturing the accuracy of large models through knowledge distillation techniques.
Solution Approach 2:
The patent introduces a knowledge distillation framework as an intermediary process between the large teacher model and the small student model. This intermediary mechanism transfers essential knowledge and patterns from the complex teacher model to the simplified student model, enabling the student model to achieve high accuracy without requiring large model size.
2Measurement precision
If more computational resources are allocated to model processing, then model accuracy is improved, but energy consumption increases
Solution Approach 1:
The patent performs computationally intensive model training and knowledge distillation in advance on powerful servers before deploying the optimized student model to mobile devices. This preliminary action transfers the high energy consumption tasks to server infrastructure, allowing the mobile device to run the deployed model with minimal energy consumption while still achieving high accuracy.
Solution Approach 2:
The patent replaces the mechanical computation approach of running large models directly on mobile hardware with a knowledge-based approach. Instead of computing complex transformations in real-time on the device, the system pre-computes and distills knowledge patterns that can be applied with simpler, lower-energy computations during inference on mobile devices.
3Adaptability or versatility
If existing transfer learning methods are used to adapt models to specific scenarios, then scenario adaptability is improved, but model performance degrades due to domain distribution differences
Solution Approach 1:
The patent applies local quality by performing scenario-specific adaptation through knowledge distillation rather than general transfer learning. The student model learns scenario-specific patterns and characteristics from the teacher model through targeted distillation processes, allowing the model to maintain high performance while adapting to specific application scenarios like autonomous driving or mobile photography.
Data Source
Figure 1
Figure 2a
Figure 2b
AI summary
This application discloses a scenario sensing method and a related device thereof, to obtain a relatively accurate sensing result of a target scenario. The method in this application includes: when scenario sensing needs to be performed on the target scenario, image data of the target scenario and/or point cloud data of the target scenario may be first obtained, and the image data and/or the point cloud data may be input into a target model; next, the target model may first perform feature extraction on the image data and/or the point cloud data to obtain a first BEV feature of the image data and/or a second BEV feature of the point cloud data; then, the target model may perform feature fusion on the first BEV feature and/or the second BEV feature based on a preset third BEV feature to obtain a feature fusion result; and finally, the target model may perform further processing on the feature fusion result to obtain a sensing result of the target scenario. In this way, the scenario sensing for the target scenario is completed.