Multimodal Question Answering via Filtered Back-Translation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems struggle to automate the complex reasoning-based queries in chart question answering, particularly for multimodal question answering, due to the lack of fine-grained perception capabilities and reliance on specific image formats, leading to errors in visual element extraction and downstream inference discrepancies.
Innovation Solution
A method and system for weakly-supervised multimodal question answering using filtered iterative back-translation, which involves a computer vision pipeline for data extraction, iterative training of semantic parser models, and probabilistic context-free grammar to synthesize SQL programs for Natural Language queries, enabling accurate answer generation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If existing end-to-end approaches use CNNs and LSTMs to extract image and question features, then the system can process chart images and answer queries, but the approach provides results only on datasets with relatively simpler queries and fails to handle complex reasoning-based queries effectively
Solution Approach 1:
The system segments the chart image into discrete visual elements (bars, lines, dots, text labels, axes) and processes them separately through dedicated detection modules. This segmentation allows the system to handle complex queries by combining results from multiple element detections rather than processing the entire image as a single unit, thereby improving both versatility and reliability.
Solution Approach 2:
The patent introduces an intermediary structured representation layer between image input and final answer output. Visual elements are detected and organized into a structured format that captures relationships between elements, serving as an intermediary representation that enables complex reasoning before generating final answers. This intermediary structure allows the system to handle diverse query types reliably.
2Ease of manufacture
If existing techniques expect chart images to be in specific formats (JavaScript or bitmaps) for extraction, then the extraction process can proceed, but the system lacks fine-grained perception capabilities and errors propagate to downstream inference tasks
Solution Approach 1:
The system employs multiple specialized detection modules that can process various chart image formats (PNG, JPG, BMP, SVG) universally. Each module (bar detector, line detector, dot detector, text detector) is designed to work across different formats, making the extraction process robust and format-agnostic while maintaining high precision through specialized detection algorithms for each visual element type.
3Measurement precision
If the system performs fine-grained perception of visual elements (chart title, X/Y-ticks, X/Y-labels, legend preview, legend labels, bar/line/dot detection), then extraction precision improves, but device complexity and processing requirements increase substantially
Solution Approach 1:
The system divides the complex extraction task into multiple specialized detection modules, each responsible for a specific visual element type (bars, lines, dots, text). This segmentation reduces the complexity of each individual module while achieving high overall precision through the coordinated work of multiple specialized components.
Solution Approach 2:
The system employs dynamic detection strategies where the activation and configuration of detection modules can be adjusted based on the input chart type and query requirements. This dynamic approach allows the system to maintain high precision while reducing unnecessary processing complexity for simpler charts.
4Device complexity
If existing approaches focus on structural, visual, relational, or simple data retrieval queries, then the extraction and answering process is simpler, but the system cannot handle complex reasoning-based queries involving perception, language understanding, and reasoning combination
Solution Approach 1:
The patent introduces a structured visual element representation as an intermediary layer between image input and query answering. This structured representation captures spatial relationships, visual properties, and semantic information in a format that enables both simple data retrieval and complex reasoning queries to be handled through the same processing framework, thereby increasing query type coverage without proportionally increasing complexity.
Data Source
AI summary
This disclosure relates generally to program synthesis for weakly-supervised multimodal question answering using filtered iterative back-translation (FIBT). Existing approaches for chart question answering mainly address structural, visual, relational, or simple data retrieval queries with fixed-vocabulary answers. The present disclosure implements a two-stage approach where, in first stage, a computer vision pipeline is employed to extract data from chart images and store in a generic schema. In second stage, SQL programs for Natural Language (NL) queries are generated in dataset by using FIBT. To adapt forward and backward models to required NL queries, a Probabilistic Context-Free Grammar is defined, whose probabilities are set to be inversely proportional to SQL programs in training data and sample programs from it. Compositional similarity-based filtration strategy employed on the NL queries generated for these SQL programs enables synthesizing, filtering, and appending NL query-SQL program pairs to training data, iteratively moving towards required NL query distribution.