Vision-Language Architecture for Flowchart Question Answering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current document AI systems lack the ability to effectively analyze and understand graphical elements in flowcharts, limiting their capacity for automatic information retrieval and question-answering capabilities.
Innovation Solution
A system utilizing a synthetic dataset generation module to create flowchart images with accompanying question-answer pairs, trained using a vision-language architecture that combines BERT and ViT for visual feature extraction and answer classification, enabling the generation of semantically meaningful question-answer pairs and supporting multiple-choice question answering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If current document AI systems are used to analyze flowcharts, then basic text processing is possible, but the ability to effectively analyze and understand graphical elements is lacking
Solution Approach 1:
The patent combines BERT (Bidirectional Encoder Representations from Transformers) for text processing with ViT (Vision Transformer) for visual feature extraction into a unified vision-language architecture. This merging enables the system to simultaneously process both textual questions and visual flowchart elements, resolving the contradiction between basic text processing capability and graphical element understanding by integrating both functionalities into a single model.
Solution Approach 2:
The patent introduces a synthetic dataset as an intermediary training resource that bridges the gap between text-based AI and visual flowchart analysis. This dataset, containing rendered flowchart images with accompanying question-answer pairs, serves as a mediator that enables the vision-language architecture to learn both visual and linguistic representations, thereby enhancing graphical element understanding while maintaining text processing capabilities.
2Measurement precision
If a vision-language architecture combining BERT and ViT is used, then visual feature extraction and answer classification improve, but computational complexity increases
Solution Approach 1:
The patent performs preliminary action by pre-training the vision-language architecture on a large synthetic dataset before deploying it for actual flowchart analysis. This pre-training phase establishes the foundational capabilities of the model, allowing it to effectively process visual and textual inputs. By preparing the model in advance with comprehensive training data, the system achieves high question-answering accuracy while the complex architecture is already optimized and ready for deployment.
3Productivity
If a synthetic dataset is generated for training, then model training efficiency improves and bias is reduced, but data generation time and computational resources increase
Solution Approach 1:
The patent implements self-service by creating a self-generating synthetic dataset pipeline that automatically renders flowchart images and generates corresponding question-answer pairs without requiring manual annotation. The system uses programmatic approaches to create training data, eliminating the need for human experts to manually create datasets. This self-service mechanism significantly improves model training efficiency and reduces bias while the automated nature of data generation minimizes the time and computational resource overhead.
Data Source
AI summary
Aspects of the disclosure include methods, systems, and computer program products for generating semantically meaningful question-answer pairs for graph-like charts, such as flowcharts. In one example, a method of implementing a Question Answering (QA) system may comprise generating a synthetic dataset of graph-like chart images. The generating may comprise rendering a plurality of graph-like chart images from a plurality of associated graph data, generating a plurality of question-answer pairs for each of the graph-like chart images, and calculating a plurality of ground truth annotations for each of the plurality of question-answer pairs and associated graph-like chart images from the plurality of associated graph data. The method of implementing the QA system may further comprise training a vision-language architecture on the synthetic dataset to answer questions about the graph-like chart images.