Visual Question Answering with Knowledge Graph Integration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Visual question answering systems are limited in their ability to provide answers that require information beyond what is directly obtainable from an image, restricting the scope of questions they can effectively answer.
Innovation Solution
Integration of knowledge graphs into visual question answering systems, utilizing neural networks for image feature extraction, question feature extraction, and fusion of these features with knowledge graph representations to generate answers, where a classification engine assigns probabilities to nodes in the knowledge graph to determine the answer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If visual question answering systems rely only on image data, then the system complexity remains manageable, but the scope of answerable questions is limited
Solution Approach 1:
The patent merges visual question answering capabilities with knowledge graph processing by integrating a knowledge graph processing module into the existing VQA architecture. This allows the system to combine image features with external knowledge from the knowledge graph, expanding the scope of answerable questions while maintaining a unified system structure that manages complexity through modular integration
2Adaptability or versatility
If knowledge graphs are integrated into visual question answering systems, then the scope of questions that can be answered expands, but the device complexity increases
Solution Approach 1:
The patent segments the VQA system into distinct functional modules: an image processing module that extracts visual features, a knowledge graph processing module that retrieves and processes external knowledge, and a fusion module that integrates both information streams. This segmentation allows each component to be optimized independently while managing overall system complexity through clear separation of concerns
3Adaptability or versatility
If additional knowledge sources are leveraged, then the breadth of queries that can be addressed increases, but the processing requirements and system complexity increase
Solution Approach 1:
The patent introduces a knowledge graph processing module as an intermediary between the image processing and answer generation components. This intermediary module retrieves relevant knowledge from external knowledge graphs, processes it appropriately, and integrates it with visual features, thereby enabling broader query coverage while managing processing complexity through a dedicated intermediate layer
Data Source
AI summary
Aspects of the current subject matter are directed to a system in which knowledge graphs are incorporated with visual question answering. A knowledge graph is integrated into a visual question answering system to provide additional knowledge from one or more sources to answer a question about an image. Aspects of the current subject matter are directed to a neural network approach that combines methods of image feature extraction and questions processing with a neural network, such as a graph neural network, that operates on knowledge graphs. The graph neural network takes input vector representations of the nodes as inputs and combines them according to their relationships into question-specific representations. The question-specific representations are then processed with the image features and the question features to generate an answer.