Collaborative BEV Fusion for Self-Driving Trajectory Decisions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing self-driving large language models lack reasoning learning based on collaborative perception features, leading to accuracy issues in trajectory decision-making, and existing collaborative perception algorithms face challenges with attention drift and high processing costs.
Innovation Solution
A method involving the fusion of RGB images and point cloud data from multiple viewpoints using feature extraction networks and deformable cross-attention mechanisms, followed by BEV feature fusion and integration with a large vision language model for trajectory decision-making.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If attention mechanism is used to aggregate features from consecutive video frames, then video comprehension is enhanced, but processing cost increases significantly and attention drift problem occurs
Solution Approach 1:
The patent extracts and removes the problematic attention mechanism from the video processing pipeline. Instead of using attention mechanisms to aggregate features from consecutive frames, the invention processes each frame independently through the same feature extraction network, eliminating attention drift and reducing processing complexity while maintaining comprehension accuracy.
Solution Approach 2:
The patent segments the video processing into independent frame-level operations. Each video frame is processed separately through identical feature extraction networks without cross-frame attention mechanisms, dividing the complex temporal aggregation problem into simpler, independent spatial feature extraction tasks that can be parallelized.
2Measurement precision
If collaborative perception features are integrated into large language models, then trajectory decision-making accuracy is improved, but system complexity increases
Solution Approach 1:
The patent merges collaborative perception features directly into the large language model's input processing pipeline. The BEV features from multiple vehicles and road-side units are fused and fed into the LLM alongside prompt information, creating a unified decision-making system that improves trajectory accuracy without requiring separate complex reasoning modules.
Solution Approach 2:
The patent introduces a feature fusion module as an intermediary between the collaborative perception system and the large language model. This mediator processes and standardizes the multi-source BEV features before presenting them to the LLM, simplifying the integration complexity while maintaining the benefits of collaborative perception for trajectory decision-making.
3Measurement precision
If multi-sensor information fusion is performed, then perception precision and range are improved, but processing time increases
Solution Approach 1:
The patent performs preliminary feature extraction and BEV transformation on multi-sensor data from all sources (ego vehicle, surrounding vehicles, road-side units) before the decision-making phase. By pre-processing and fusing sensor features into standardized BEV representations in advance, the system reduces real-time processing time while maintaining high perception precision for trajectory decisions.
Data Source
AI summary
A method and an apparatus for collaborative end-to-end large model-oriented self-driving trajectory decision-making are provided. The method includes: processing an RGB image from ego vehicle, an RGB image from surrounding vehicle, and an RGB image from road-side by using a first feature extraction network to obtain a first image feature, a second image feature, and a third image feature, respectively; fusing first image feature, second image feature, and third image feature to obtain an image fusion feature; processing point cloud data of road-side to obtain a road-side point cloud feature; processing image fusion feature and road-side point cloud feature to obtain a first BEV feature and a second BEV feature; fusing first BEV feature and second BEV feature to obtain a fused BEV feature; and fusing prompt information and fused BEV feature to obtain text information, and then processing text information to obtain an ego vehicle trajectory decision-making result.