Visual Dialogue System Scene Graph Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing visual dialogue methods struggle to accurately parse long or complex questions and corresponding answers, especially in complex scenes with numerous interactions between foreground objects and noisy backgrounds, leading to information redundancy and ineffective visual reasoning.
Innovation Solution
The proposed visual dialogue method involves preprocessing text and image data to obtain feature sequences, constructing a text corpus, obtaining text sequence knowledge using a potential knowledge searcher, building a sparse scene graph, and performing data fusion using attention mechanisms and graph convolution to enhance dialogue accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If an encoder is used to extract global features and fuse them into a joint representation, then the visual dialogue system can process multiple inputs, but information redundancy occurs and visual co-reference resolution fails
Solution Approach 1:
The patent segments the visual dialogue processing into distinct modules: a visual question answering module that handles image-question pairs, and a dialogue generation module that processes dialogue history. This segmentation allows each module to specialize in specific tasks, avoiding information redundancy while maintaining the ability to process multiple inputs including images, questions, and dialogue history.
2Measurement precision
If attention mechanism or graph model is used to mine necessary visual content, then visual co-reference resolution is avoided, but the method is powerless to deal with complex scenes
Solution Approach 1:
The patent introduces a scene graph as an intermediary representation that captures complex relationships between objects in the image. The scene graph serves as a mediator between the visual input and the dialogue generation, enabling the system to reason about complex scenes by representing objects, their attributes, and relationships in a structured format that can be easily processed by the attention mechanism.
3Reliability
If knowledge-based methods are used to improve reasoning capability, then single knowledge source is utilized, but parsing of long and difficult sentences remains ineffective
Solution Approach 1:
The patent merges multiple knowledge sources including visual knowledge from the image, contextual knowledge from dialogue history, and linguistic knowledge from the question. By combining these different knowledge sources in the joint representation, the system can effectively parse long and difficult sentences while maintaining strong reasoning capability, as each knowledge source compensates for the limitations of others.
Data Source
AI summary
A visual dialogue method and system is provided. The method includes obtaining original input data, where the original input data includes current image data and a new question, and the new question is related to the current image data; preprocessing text data and image data in the original input data to obtain a text feature sequence and a visual feature sequence, respectively; using a VisDial dataset to construct a text corpus; obtaining text sequence knowledge by using a potential knowledge searcher based on the visual feature sequence and the text corpus; constructing a sparse scene graph based on the visual feature sequence; performing data fusion on the text feature sequence, the visual feature sequence, the text sequence knowledge, and the sparse scene graph to obtain a data fusion result; and obtaining dialogue content of the new question by using a decoder based on the data fusion result.