Tri-Modal Image Region Prediction for Speech-Designated Places
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems fail to effectively identify a designated place in an image based on a user's utterance, despite advancements in object classification using sensor data and image data.
Innovation Solution
An information processing apparatus utilizing a trained machine learning model with an encoder and decoder to process text, visual, and segmentation mask features, allowing for the prediction of regions in an image corresponding to user instructions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If object classification is performed using sensor data and image data, then object identification capability is improved, but the ability to identify designated places based on user utterances remains insufficient
Solution Approach 1:
The patent combines three types of data processing: text information processing (utterance), image data processing (visual features), and segmentation data processing (region division). These are merged through a multi-modal encoder that processes all three inputs simultaneously and feeds them to a decoder that outputs the designated region, thereby achieving both precise object identification and utterance-based place identification
Solution Approach 2:
The machine learning model is designed to perform multiple functions: it processes text utterances to understand user intentions, processes images to extract visual features, processes segmentation masks to identify regions, and integrates all three to output the designated place. This multi-functional design enables the system to handle both object classification and utterance-based navigation tasks
2Productivity
If traditional image processing is used, then processing speed is maintained, but the ability to understand and process multi-modal information (text, image, segmentation) is insufficient
Solution Approach 1:
The patent segments the processing task into distinct modules: a text encoder that processes utterances, an image encoder that extracts visual features, and a segmentation encoder that processes region division data. Each module processes its specific input independently and efficiently, then the results are combined by the decoder to produce the final output, maintaining processing speed while enabling multi-modal understanding
Data Source
Figure 1A~1B
Figure 2
Figure 3
AI summary
An information processing apparatus in embodiments performs at least one trained machine learning model that includes an encoder and a decoder. The encoder receives inputs of text information including designation of a place, a first image that is an image captured by an image capturing apparatus and that includes the place, and a second image obtained by dividing a region for every identical object in the first image, and outputs tri-modal features that have been generated to include visual features of the first image that has been captured, visual features of the second image obtained by dividing the region, and language features of the text information. The decoder outputs a region on the first image corresponding to the designation of the place in the text information, by using the tri-modal features.