Audio-Visual Question Answering With Balanced Cross-Modal Attention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AVQA machine learning models are biased due to imbalanced training datasets, leading to a reliance on common answers and neglecting the importance of audio-visual reasoning.
Innovation Solution
A machine learning model with an audio-spectrogram-transformer (AST) branch and cross-modal pixel-wise attention is trained on a balanced MUSIC-AVQA v2.0 dataset, incorporating additional real videos to enrich audio-visual relationships, and employs spatial and temporal grounding to enhance audio-visual fusion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If models are trained on existing AVQA datasets, then training can proceed with available data, but the models develop bias toward common answers and fail to perform genuine audio-visual reasoning
Solution Approach 1:
The patent inverts the training approach by creating a balanced dataset where previously underrepresented answer categories are oversampled. Instead of accepting the natural bias in existing data, the methodology actively reverses the distribution imbalance by sampling more examples from minority classes, thereby correcting the model's tendency to favor common answers and enabling genuine reasoning about less frequent audio-visual relationships.
Solution Approach 2:
The patent changes the data distribution parameter by applying stratified sampling with oversampling of minority classes. This transforms the training data composition from heavily skewed toward common answers to a balanced distribution across all answer categories, allowing the model to learn diverse audio-visual relationships rather than memorizing frequent patterns.
2Ease of manufacture
If models rely on common answers from imbalanced datasets, then training is simpler, but the models neglect audio-visual reasoning and produce biased results
Solution Approach 1:
The patent performs preliminary data balancing before model training by applying stratified sampling and oversampling techniques to the training dataset. This preprocessing step ensures that the model encounters diverse answer categories early in training, preventing it from developing biases toward common answers and establishing robust audio-visual reasoning capabilities from the outset.
3Loss of information
If datasets are enriched with additional real videos, then audio-visual relationships are better captured, but data collection and processing complexity increases
Solution Approach 1:
The patent uses video augmentation techniques to create synthetic copies and variations of existing video data. By applying transformations such as temporal cropping, frame sampling, and audio-visual pairing variations, the methodology generates additional training examples from limited source material, enriching the dataset diversity without requiring proportional increases in manual data collection and annotation efforts.
Data Source
AI summary
The present disclosure describes techniques for improving audio-visual question answering. A machine learning model is configured for audio-visual question answering (AVQA). The machine learning model comprises a first sub-model configured to capture semantic audio information and output an audio spatial feature map xas(1). The machine learning model comprises a second sub-model configured to extract visual features xvs and audio features xas and further configured to obtain a question vector xq. The machine learning model comprises a third sub-model configured to capture audio-visual correspondence at a granular level. A balanced AVQA dataset is created. The balanced AVQA dataset comprises balanced answer distribution in each question category. The machine learning model is trained to answer questions about visual objects, sounds, and their associations in videos using at least a subset of the balanced AVAQ dataset.