LLM Finetuning with Scene Graphs for Visual Navigation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing large language models are unreliable for visual navigation tasks due to the lack of reliable training data and the natural ambiguity and incomplete information present in real-world environments, making it challenging to navigate to a desired object effectively.
Innovation Solution
A method is employed to finetune a language model using scene graphs derived from 3D scene data, generating natural language prompts and ground truth paths to create targeted training examples, leveraging a pretrained model for improved performance in visual navigation tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a language model is trained for visual navigation tasks using real-world data, then the model can handle natural ambiguity and incomplete information, but the training data is rare and insufficient for effective model performance
Solution Approach 1:
The patent creates synthetic training data by copying and transforming existing 3D scene data into structured scene graph representations. This synthetic data mirrors real-world navigation scenarios while being systematically generated, thereby providing sufficient training material without requiring actual real-world navigation datasets.
Solution Approach 2:
The patent performs preliminary processing of 3D scene data to construct scene graphs and generate navigation paths before the actual model training. By pre-computing the ground truth navigation paths and structuring the data in advance, the system prepares comprehensive training material that would otherwise require extensive real-world data collection.
2Quantity of substance
If a language model is finetuned using synthetic scene graph data, then the training data requirements are reduced, but the model must learn to navigate natural ambiguity and incomplete information in real environments
Solution Approach 1:
The patent introduces scene graphs as an intermediary representation layer between the synthetic training data and the actual navigation task. The scene graph structure serves as a mediator that organizes 3D scene information into a format that captures spatial relationships and navigation logic, enabling the model to learn from synthetic data while maintaining the ability to handle real-world ambiguity.
Solution Approach 2:
The patent applies different levels of detail and structure to different parts of the scene graph based on their importance for navigation. Critical navigation elements receive more detailed representation while less important elements are summarized, allowing the model to focus on learning essential navigation patterns from synthetic data while being prepared for the variability of real environments.
Data Source
AI summary
A method for finetuning a pretrained language model for performing a visual navigation task is described. Scene data is provided that describes a plurality of scenes. Scene graphs that represent the plurality of scenes are derived based on the scene data. For different combinations of a given starting room and target object, a ground truth shortest path from the starting room to the target object in the scene is determined based on the scene graph. Based on the scene data and the scene graphs, natural language prompts are derived that prompt the language model to predict, given a current room and the target object in the scene, a next navigation step of a shortest path from the starting room to the target object in the scene. Together, the natural language prompts and the ground truth shortest paths are used to finetune the pretrained language model.