Object-Level Tokenization for Autonomous Driving LLM Planning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
End-to-end autonomous driving models suffer from severe performance degradation in long-tail scenarios due to data scarcity and inefficient tokenization, failing to leverage the common-sense reasoning abilities of Large Language Models (LLMs) effectively.
Innovation Solution
A Multi-Modal Large Language Model (MM-LLM) with tokenized object-level knowledge is developed, utilizing a pre-trained tokenizer to tokenize visual input into object-level tokens, aligned using an adapter, and trained with perception, reasoning, and planning question-answering pairs to enhance scene understanding and planning capabilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If detailed textual scene descriptions are used as prompts, then the LLM can leverage common-sense reasoning abilities, but the performance becomes highly reliant on prompt quality and detail requiring extensive engineering
Solution Approach 1:
The scene description is segmented into structured object-level tokens with specific attributes (type, position, velocity, etc.) rather than using unstructured detailed text. This segmentation allows the LLM to receive organized information that triggers reasoning capabilities without requiring complex prompt engineering to extract details.
Solution Approach 2:
A scene tokenizer is introduced as an intermediary component that transforms raw sensory inputs into structured object-level tokens. This mediator handles the complexity of information extraction and organization, allowing the LLM to focus on reasoning tasks without being burdened by prompt engineering complexity.
2Ease of operation
If simple heuristic prompts are used, then the prompt engineering is simplified, but the LLM cannot tap into common-sense reasoning abilities due to insufficient scene understanding
Solution Approach 1:
The scene tokenizer performs preliminary processing of sensory inputs into structured object-level tokens before presenting them to the LLM. This preliminary action ensures that scene understanding is established beforehand, allowing the LLM to leverage its reasoning abilities with simple prompts rather than requiring complex prompt engineering to compensate for poor scene understanding.
3Adaptability or versatility
If pre-trained encoders with querying transformers are used to tokenize features, then the model can process sensory inputs, but training requires billions of question-answer pairs which are not available for autonomous driving
Solution Approach 1:
Instead of requiring complete billion-scale QA pairs for training, the approach uses a partial dataset of one million QA pairs focused on object-level scene understanding. The structured tokenization framework allows the model to achieve effective scene processing capabilities with this reduced training data quantity by focusing on essential object attributes rather than exhaustive scene descriptions.
4Quantity of substance
If current MM-LLM datasets with fewer than one million QAs are used, then the training data requirement is reduced, but the models exhibit poor performance in reasoning and planning tasks due to lack of scene understanding and grounding capability
Solution Approach 1:
The approach changes the parameters of the training data by focusing on structured object-level attributes (type, position, velocity, acceleration, etc.) rather than unstructured textual descriptions. This parameter change in how scene information is represented allows the model to achieve better reasoning performance with fewer training examples, as the structured format provides explicit grounding information that triggers LLM reasoning capabilities.
Data Source
AI summary
Apparatuses, systems, and techniques for enhancing autonomous driving systems. In at least one embodiment, visual input corresponding to an observable environment is tokenized into object-level knowledge and provided to a large language model (LLM). Object-level tokens are processed by the LLM to enhance autonomous vehicle route-planning, reducing trajectory error and decreasing collision rates.