Tri-Modal Image Region Prediction for Utterance-Guided Navigation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems fail to accurately identify a designated place in an image based on user utterance, lacking integration of utterance data for object classification and image recognition.
Innovation Solution
An information processing apparatus utilizing a trained machine learning model with an encoder and decoder to fuse visual and language features through attention mechanisms, enabling the prediction of regions in an image corresponding to user utterances.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If object classification using point group data and image data is implemented, then autonomous driving capability is improved, but the ability to identify designated places by utterance is insufficient
Solution Approach 1:
The image is divided into multiple regions corresponding to different objects, and separate encoding is performed for each region. This segmentation allows the system to focus on specific areas mentioned in the utterance, improving the precision of designated place identification while maintaining autonomous driving capabilities.
Solution Approach 2:
An attention mechanism is introduced as an intermediary to weight and select relevant visual features from different image regions based on the utterance content. This mediator enables the system to bridge between general object classification and precise designated place identification by dynamically focusing on relevant regions.
2Extent of automation
If full-or semi-automatic driving using classification results is implemented, then autonomous movement technology is improved, but the technique for receiving utterance instructions and identifying designated places is lacking
Solution Approach 1:
The system integrates multiple functions into a unified framework: object classification, utterance processing, and designated place identification are performed within the same neural network architecture. This multi-functional approach allows the moving object to maintain autonomous driving capabilities while adding versatile utterance-based navigation without requiring separate specialized systems.
Solution Approach 2:
The system adds a new dimension of text information processing to the existing visual processing framework. By incorporating language features from utterances as an additional input modality and fusing them with visual features through attention mechanisms, the system enables utterance-based designated place identification while preserving existing autonomous movement capabilities.
3Measurement precision
If tri-modal feature fusion using attention mechanism is implemented, then designated place identification accuracy is improved, but computational complexity increases
Solution Approach 1:
The computational process is segmented into distinct encoding stages for different modalities (visual features from the first image, visual features from the second divided image, and language features from text information). This segmentation allows for efficient parallel processing of each modality independently before fusion, reducing overall computational complexity while maintaining high identification accuracy.
Solution Approach 2:
The attention mechanism applies partial action by selectively focusing computational resources only on relevant features and regions. Instead of processing all possible feature combinations, the attention mechanism weights and selects only the most relevant visual and language features for the specific utterance, reducing unnecessary computations while achieving high precision in designated place identification.
Data Source
AI summary
An information processing apparatus in embodiments performs at least one trained machine learning model that includes an encoder and a decoder. The encoder receives inputs of text information including designation of a place, a first image that is an image captured by an image capturing apparatus and that includes the place, and a second image obtained by dividing a region for every identical object in the first image, and outputs tri-modal features that have been generated to include visual features of the first image that has been captured, visual features of the second image obtained by dividing the region, and language features of the text information. The decoder outputs a region on the first image corresponding to the designation of the place in the text information, by using the tri-modal features.