Visio-Linguistic Reasoner Using Knowledge Graph Embeddings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for visio-linguistic understanding require significant computational power and time due to the need for pre-training contextual visio-linguistic reasoners on large image and text datasets, making them inefficient for handling variations in image data and downstream tasks.
Innovation Solution
The method employs a contextual language model reasoner, converting image and text inputs into a compatible format using deep learning models and external knowledge graphs to generate contextual embeddings, bypassing the need for pre-training and reducing computational requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If pre-training is performed on large image and text datasets to create contextual visio-linguistic reasoners, then understanding capability is improved, but computational power and time requirements increase significantly
Solution Approach 1:
The patent extracts and utilizes pre-trained contextual language model reasoners (such as BERT, RoBERTa, XLNet) that were previously trained on large text datasets, and adapts them for visio-linguistic tasks by integrating image region proposal vectors. This extraction approach avoids the need to pre-train from scratch on large image-text datasets, significantly reducing computational power and time requirements while maintaining understanding capability.
2Productivity
If domain-specific architectures with simple attention are used to associate image region proposal vectors and word embeddings, then computational efficiency is improved, but adaptability to different tasks decreases
Solution Approach 1:
The patent employs universal contextual language model reasoners that can handle multiple visio-linguistic tasks through a unified framework. By integrating image region proposal vectors with the pre-trained language model's attention mechanisms, the system achieves both computational efficiency and adaptability across different tasks such as visual question answering, image captioning, and visual entailment, without requiring task-specific architecture modifications.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
This disclosure relates generally to visio-linguistic understanding. Conventional methods use contextual visio-linguistic reasoner for visio-linguistic understanding which requires more compute power and large amount of pre-training data. Embodiments of the present disclosure provide a method for visio-linguistic understanding using contextual language model reasoner. The method converts the visual information of an input image into a format that the contextual language model reasoner understands and accepts for a downstream task. The method utilizes the image captions and confidence score associated with the image captions along with a knowledge graph to obtain a combined input in a format compatible with the contextual language model reasoner. Contextual embeddings corresponding to the downstream task is obtained using the combined input. The disclosed method is used to solve several downstream tasks such as scene understanding, visual question answering, visual common-sense reasoning and so on.