Grounded Visual Question Answering via Dynamic Two-Level Feature Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing Visual Question Answering (VQA) systems lack the ability to provide verifiable visual evidence for their answers, making them less reliable and limiting their application in scenarios like assisting visually impaired individuals.
Innovation Solution
A grounded visual question answering method based on dynamic two-level visual information fusion, which integrates a problem-guided regional-level dynamic multi-scale method and a cross-modal multi-scale fusion module to generate both text answers and image grounding masks, enhancing the system's reliability and versatility.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional VQA methods output only text answers, then the system is simple to implement, but the reliability and verifiability of answers deteriorates
Solution Approach 1:
The patent merges the text answer generation task with the visual evidence localization task into a unified VQA framework. The model simultaneously outputs both the text answer and the corresponding grounding mask, allowing verification of answer reliability through visual evidence while maintaining a cohesive system architecture rather than separate independent modules
2Adaptability or versatility
If fixed-resolution masks are used for answer grounding, then the implementation is simple, but the adaptability to different answer types and complexities deteriorates
Solution Approach 1:
The patent implements dynamic multi-scale feature fusion that adapts mask resolution based on the specific VQA task requirements. The model dynamically selects and fuses features at different resolution levels (1/4, 1/8, 1/16 of original image size) depending on the complexity and type of answer needed, allowing flexible adaptation without predetermined fixed resolution constraints
3Measurement precision
If detailed visual grounding masks are generated, then the answer verifiability is improved, but the computational overhead and processing time increases
Solution Approach 1:
The patent segments the visual processing into multiple hierarchical feature levels with different resolutions. By processing visual information at segmented resolution levels (coarse to fine) and selectively fusing features based on task requirements, the model achieves precise grounding masks while distributing computational load across different processing stages rather than requiring full high-resolution processing for all cases
Data Source
AI summary
A ground visual question-answering method based on dynamic dual-level visual information fusion includes using a dual-level multiscale network, which is divided into language-guided pixel-level features and region-level features. These two scale branches are combined to predict the final textual answer and ground answer. Furthermore, a question-guided dynamic region-level feature localization network is proposed to locate visual information guided by the question and adaptively assign masks of different sizes to ground answers, thereby enhancing the accuracy of locating and segmenting small targets. Additionally, a cross-modal aggregation module is designed to fuse features from both levels, enhancing the fusion of pixel-level and region-level features to improve the segmentation effect of ground answer masks' edges. The ground visual question-answering system built by the language-guided adaptive dual-level feature fusion network in this invention can effectively improve the accuracy of the entire model while answering questions and generating answer ground masks simultaneously.