Visual Question Answering Model Training with Synthetic Data Augmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Visual question answering (VQA) models face challenges in robustness and generalization due to overfitting when trained on small target datasets, especially in specialized domains like medical or art, where collecting additional annotations is expensive and time-consuming.
Innovation Solution
A method that generates synthetic question-answer pairs for unlabeled images using a teacher model, which are then merged with real data to create a self-augmented training set, allowing a student model to be trained for improved robustness and generalization, without requiring pre-trained object detectors or handcrafted rules.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If a VQA model is trained on a small target dataset, then training cost and time are reduced, but the model suffers from overfitting and reduced generalization performance
Solution Approach 1:
The method performs preliminary data augmentation by generating synthetic question-answer pairs before training the student model. A teacher model is first trained on the small target dataset, then used to generate synthetic training data that is merged with real data to create an augmented training set, thereby preparing enhanced training data in advance to prevent overfitting
Solution Approach 2:
The method creates synthetic copies of training data by using the teacher model to generate pseudo-labeled question-answer pairs for unlabeled images. These synthetic data copies are then merged with real labeled data to form an augmented training set, allowing the student model to learn from diverse examples without requiring additional manual annotations
2Reliability
If additional annotations are collected to expand the dataset, then model robustness and generalization improve, but cost and time requirements increase prohibitively
Solution Approach 1:
The system performs self-service data augmentation by using the trained teacher model to automatically generate synthetic question-answer pairs for unlabeled images. This self-generated data serves as additional training material, eliminating the need for manual annotation while expanding the effective training dataset size
Solution Approach 2:
The method changes the state of unlabeled images by transforming them into pseudo-labeled training samples through the teacher model. This parameter transformation converts unusable unlabeled data into valuable training data, effectively expanding the dataset without manual intervention
Data Source
AI summary
Systems and methods for training a visual question answer model include training a teacher model by performing image conditional visual question generation on a visual language model (VLM) and a targeted visual question answer dataset using images to generate question and answer pairs. Unlabeled images are pseudolabeled using the teacher model to decode synthetic question and answer pairs for the unlabeled images. The synthetic question and answer pairs for the unlabeled images are merged with real data from the targeted visual question answer dataset to generate a self-augmented training set. A student model is trained using the VLM and the self-augmented training set to return visual answers to text queries.