Attention Memory Fusion for Visual Dialog Reference Resolution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Artificial Neural Networks (ANNs) trained for visual question answering struggle to resolve semantic references, limiting their ability to interpret follow-up questions that refer to previous portions of a dialog, as they cannot accurately understand pronouns and other semantic references.
Innovation Solution
A dialog engine with an attention module that generates a fused attention map by combining a tentative attention map with a relevant attention map, allowing the network to resolve semantic references and generate accurate responses to questions that include ambiguous terms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a previous ANN architecture is used for visual question answering, then simple questions about image content can be answered, but semantic references in questions cannot be resolved
Solution Approach 1:
The system performs preliminary action by generating and storing an attention map during the encoding phase that captures the relationship between image regions and previous dialog context. This pre-computed attention information is then retrieved and fused with new query attention to resolve semantic references, allowing the system to handle pronouns and references without re-processing the entire dialog history.
Solution Approach 2:
The attention map serves as an intermediary that bridges the gap between the image content and the dialog context. By computing attention weights that link image regions to previous questions and answers, the system creates a mediator representation that enables semantic reference resolution. This attention mechanism acts as the intermediary layer that connects visual features with linguistic context.
2Measurement precision
If attention maps are fused to resolve semantic references, then accuracy in answering follow-up questions improves, but computational complexity increases
Solution Approach 1:
The system extracts only the necessary attention information from previous dialog turns by retrieving relevant attention maps based on the current query context. Instead of processing all previous interactions, the system extracts and fuses only the attention maps that are relevant to the current semantic reference, reducing the computational burden while maintaining accuracy.
Solution Approach 2:
The system applies partial action by fusing attention maps selectively rather than combining all possible historical attention information. The attention retrieval mechanism identifies and fuses only the necessary portion of historical attention maps that are relevant to the current query, avoiding the excessive computation that would result from processing the entire dialog history.
Data Source
AI summary
A dialog engine configured to answer a sequence of questions related to an image. An attention module residing in the dialog engine includes an attention retrieval process and a tentative attention process. The attention retrieval process retrieves a relevant attention map that may have previously been used to answer a previous query. The tentative attention process generates a tentative attention map based on an image and other input parameters. The attention module combines the relevant attention map and the tentative attention map to generate a fused attention map. Based on the fused attention map, the dialog engine generates a response to the query. Finally, the dialog engine stores the fused attention map in an attention memory for use in answering future queries.