Autonomous Driving VLM Training With 3D Sensor-Language Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional vision language models (VLMs) trained on two-dimensional data fail to accurately perceive the three-dimensional environment surrounding autonomous vehicles, leading to inadequate assessment of object distances, sizes, and positions, and thus unsafe driving decisions.
Innovation Solution
A data generation pipeline is employed to generate training data for VLMs, including key frames, prompts, and conversations, enabling the VLMs to interpret 3D image and position data, using a projector that processes multi-view image features and 3D position encoding, and large-language models to generate driving plans.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional VLMs are trained on two-dimensional video data, then the training process is simple and data availability is high, but the perception accuracy of three-dimensional environment is insufficient
Solution Approach 1:
The patent transitions from training VLMs on 2D video data to processing 3D point cloud data from sensor arrays. This dimensional change enables the model to perceive depth, volume, and spatial relationships accurately, directly addressing the limitation of conventional 2D-based approaches while improving environmental assessment capability.
Solution Approach 2:
The patent introduces a specialized projector component that acts as an intermediary between the 3D sensor data and the VLM. This projector processes multi-view image features and 3D position encoding to generate aligned features, serving as a bridge that enables the language model to effectively interpret complex 3D spatial information.
2Reliability
If three-dimensional sensor data is used for training, then the environmental assessment capability is improved, but the availability of annotated training data is limited
Solution Approach 1:
The patent employs a data generation pipeline that pre-processes and annotates 3D sensor data before training the VLM. By preparing diverse training data in advance with proper annotations, the system overcomes the scarcity of ready-to-use 3D training data and enables effective model training for safe driving decisions.
Solution Approach 2:
The patent creates synthetic or processed copies of 3D sensor data through the data generation pipeline. This approach multiplies the available training data by generating multiple views, augmentations, and annotated versions of the original sensor data, effectively increasing the training data volume without requiring additional physical sensors or manual annotation efforts.
3Ease of operation
If language-based reasoning is incorporated into AV control, then the interpretability of driving decisions is improved, but the processing time may increase
Solution Approach 1:
The patent pre-trains the VLM on diverse 3D data and conversational datasets before deployment. This preliminary training equips the model with the reasoning capabilities needed for language-based interpretation of driving decisions, allowing it to process and explain decisions efficiently during actual operation without excessive processing delays.
Data Source
AI summary
In various embodiments, a computer-implemented method for training vision language models includes generating, based on a set of key frames that include sensor data captured during operation of a vehicle, a subset of key frames that meets a diversity criterion, generating, based on the set of key frames, a set of prompts that describe the operation of the vehicle, generating, based on the subset of key frames and the set of prompts, a set of conversations that include one or more questions and one or more corresponding answers associated with operation of the vehicle, generating training data that includes the subset of key frames, the set of prompts, and the set of conversations, and performing, based on the training data, one or more operations to train a vision language model to generate a trained vision language model.