Indoor Navigation Using Knowledge Graphs for Vision-Language Reasoning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current vision-language navigation technologies face challenges in indoor navigation, particularly in understanding abstract instructions and making reasonable action decisions due to limited cross-modal understanding and poor reasoning capabilities between visual and language modalities, leading to inefficient navigation in unknown environments.
Innovation Solution
An indoor navigation method that enhances cross-modal understanding by fusing visual and instruction features with knowledge graphs, performing multi-step graph convolutional reasoning, and using attention mechanisms to determine navigation decisions, thereby improving the capability to recognize rooms and objects and make more reasonable action decisions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If vision-language navigation model uses basic visual and language features for navigation, then the system complexity is low, but the cross-modal understanding capability and reasoning ability are insufficient
Solution Approach 1:
The patent introduces knowledge graphs as an intermediary component between visual and language modalities. The knowledge graph stores semantic relationships between objects and rooms, serving as a mediator that enhances cross-modal understanding without directly increasing the complexity of the core navigation model. This intermediary structure allows the system to leverage external knowledge bases rather than embedding all reasoning capabilities within the navigation model itself.
Solution Approach 2:
The navigation system is segmented into distinct functional modules: visual feature extraction, language feature extraction, knowledge graph reasoning, and navigation decision-making. This segmentation allows each module to be optimized independently while maintaining overall system functionality, resolving the contradiction by organizing complexity into manageable, specialized components.
2Reliability
If multi-step graph convolutional reasoning is performed on knowledge graphs, then the reasoning capability is enhanced, but the computation time and processing complexity increase
Solution Approach 1:
The knowledge graph is pre-constructed and pre-populated with semantic relationships between indoor objects and rooms before the navigation task begins. This preliminary action allows the system to perform faster lookups and reasoning during actual navigation, as the heavy lifting of organizing semantic knowledge has already been completed in advance rather than during real-time navigation.
3Measurement precision
If feature fusing and reinforcing operations are performed multiple times on knowledge graphs and visual features, then the navigation decision accuracy is improved, but the processing complexity and computational load increase
Solution Approach 1:
The patent merges multiple feature representations (visual features, language features, and knowledge graph features) into a unified feature space through feature fusing operations. This combining approach allows the system to leverage complementary information from different modalities simultaneously, improving navigation decision accuracy by creating a more comprehensive and robust feature representation that captures relationships across all input types.
Data Source
Figure 1~3
Figure 4~5
Figure 6~8
AI summary
An indoor navigation method, apparatus, equipment and storage medium are provided. An indoor navigation method, applied to an navigation equipment, including: receiving an instruction for navigation, and collecting an environment image; extracting an instruction room feature and an instruction object feature carried in the instruction, and determining a visual room feature, a visual object feature, and a view angle feature based on the environment image; fusing the instruction object feature and the visual object feature with a first knowledge graph representing an indoor object association relationship to obtain an object feature, and determining a room feature based on the visual room feature and the instruction room feature; and determining a navigation decision based on the view angle feature, the room feature, and the object feature. Through the embodiments of the present disclosure, a capability of understanding between a vision modal and a language modal is enhanced in determining a navigation action decision, and entity knowledge reasoning for the room and the object is introduced, so that the action decision in the navigation is more reasonable.