Visual Question Answering Graph Fusion for Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current visual question answering systems face challenges in effective cross-modal data fusion and relationship reasoning, leading to reduced accuracy in answering complex questions due to noise introduction and inadequate attention learning for both image and question features.
Innovation Solution
A method involving the construction of visual and question graphs using object detection networks and word embedding algorithms, followed by multimodal fusion through graph convolutional networks and attention mechanisms to align and update features, enhancing the reasoning ability for complex inputs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional visual question answering systems are used, then the system structure is simple, but the accuracy is reduced due to noise introduction and inadequate attention learning
Solution Approach 1:
The patent segments the visual question answering system into distinct graph-based modules: visual graph construction from image features, question graph construction from text features, and graph convolutional network processing. This segmentation allows each module to specialize in specific tasks, reducing noise interference and improving accuracy while maintaining manageable system complexity through modular architecture.
Solution Approach 2:
The patent introduces graph structures as intermediary representations between raw image/text inputs and final answers. The visual graph and question graph serve as mediators that organize features before processing, enabling better relationship reasoning and attention learning. This intermediary step filters noise and structures information, improving accuracy without proportionally increasing complexity.
2Reliability
If graph-based multimodal fusion is implemented, then relationship reasoning is enhanced, but computational complexity increases
Solution Approach 1:
The patent performs preliminary graph construction and feature extraction before the main reasoning process. By pre-organizing visual and textual features into graph structures with defined nodes and edges, the system prepares data in an optimized format that facilitates efficient relationship reasoning during inference, reducing computational burden during the actual Q&A process.
Solution Approach 2:
The patent employs dynamic graph convolutional networks that adaptively process the constructed graphs. The graph convolutional layers dynamically adjust feature aggregation based on graph structure and content, enabling flexible relationship reasoning. This dynamic processing enhances reliability by capturing complex relationships while managing computational complexity through adaptive rather than exhaustive processing.
Data Source
AI summary
A method for visual question answering, a computer device implementing the method and a medium for storing instructions on performing the method are provided. The method includes: acquiring an input image and an input question; constructing a visual graph based on the input image, wherein the visual graph comprises a first node feature and a first edge feature; constructing a question graph based on the input question, wherein the question graph comprises a second node feature and a second edge feature; performing a multimodal fusion on the visual graph and the question graph to obtain an updated visual graph and an updated question graph; determining a question feature based on the input question; determining a fusion feature based on the updated visual graph, the updated question graph and the question feature; and generating a predicted answer for the input image and the input question.