Language-Guided 3D Navigation With Coarse-to-Fine Visual Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods struggle to effectively navigate a 3D environment using language commands due to difficulties in understanding spatial relationships in textual instructions and integrating visual and textual cues for navigation.
Innovation Solution
A coarse-to-fine fusion method and apparatus that combines image and language inputs by extracting hierarchical visual feature maps and text features, using a fusion module to generate attention maps, followed by reinforcement learning to guide navigation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If language commands are used to instruct navigation, then ease of operation is improved, but understanding spatial relationships deteriorates
Solution Approach 1:
The patent introduces visual feature maps as an intermediary between language commands and navigation actions. The language grounding module converts text instructions into visual representations that can be processed by the navigation system, bridging the gap between linguistic commands and spatial understanding without requiring the system to directly interpret complex spatial relationships in text
Solution Approach 2:
The navigation task is segmented into multiple processing stages: language grounding, visual feature extraction, feature fusion, and navigation action generation. This segmentation allows each module to specialize in specific aspects of the problem, improving overall performance while maintaining ease of operation with language commands
2Reliability
If visual and textual features are fused, then navigation performance is improved, but device complexity increases
Solution Approach 1:
The patent merges visual feature maps and text feature maps into a unified representation through feature fusion modules. This combining approach allows the system to leverage both visual and linguistic information simultaneously, improving navigation reliability while managing complexity through integrated processing architectures
Solution Approach 2:
The feature fusion modules serve multiple functions: they integrate visual and textual features, perform attention-based weighting, and generate unified representations for navigation. This multi-functionality reduces the need for separate processing pipelines, managing device complexity while improving navigation performance
3Measurement precision
If hierarchical visual feature maps are extracted, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The patent extracts visual features at multiple hierarchical levels (different dimensions of abstraction), allowing the system to capture both fine-grained and coarse-grained spatial information. This multi-dimensional feature extraction improves measurement precision by providing rich visual representations without requiring a single overly complex processing stage
Data Source
AI summary
Disclosed are a coarse-to-fine fusion method and apparatus for virtual space navigation based on language commands. The coarse-to-fine fusion method for virtual space navigation based on language commands includes: (a) applying an input image to an encoder to extract first to nth visual feature maps having a hierarchical structure; (b) applying an instruction having a length of N to a language model to extract a text feature map; and (c) fusing each of the first to nth visual feature maps with the text feature map to generate an attention map.