Visual Speech Recognition Bias Reduction via Synthetic Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Visual speech recognition models face reduced accuracy due to uneven representation of groups in training data, leading to biased performance on under-represented groups such as older adults and children, as the training data often lacks sufficient samples across various age, gender, and cultural backgrounds.
Innovation Solution
A method is introduced to identify under-represented groups in training data and generate synthetic samples using a generative adversarial network (GAN) to supplement the training data, ensuring more diverse and balanced representation, thereby improving the model's accuracy in interpreting speech from diverse subjects.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If training data is used as-is without augmentation, then the training process is simple and fast, but the VSR model exhibits biased performance on under-represented groups
Solution Approach 1:
The patent creates synthetic copies of speech samples from over-represented groups and modifies them to represent under-represented groups. The sample generation model generates synthetic speech samples that mimic the characteristics of under-represented groups, allowing the model to learn from these artificial examples without requiring additional real-world data collection.
Solution Approach 2:
The patent transforms existing speech samples by modifying key parameters such as pitch, formant frequencies, and spectral characteristics to match the acoustic properties of under-represented groups. This parameter transformation allows over-represented samples to be converted into synthetic samples that represent different demographic groups.
2Reliability
If more diverse training data is collected from multiple groups, then model accuracy improves, but data collection time and cost increase
Solution Approach 1:
The patent performs preliminary analysis of the existing training data to identify which groups are under-represented and determines the specific acoustic characteristics needed for each group. This preliminary assessment allows the system to generate only the necessary synthetic samples rather than collecting comprehensive real data from all possible groups.
Solution Approach 2:
Instead of collecting new data from under-represented groups, the system creates synthetic copies by transforming existing samples. This copying approach bypasses the time-consuming data collection process while still providing diverse training examples for the VSR model.
3Reliability
If synthetic samples are generated to balance group representation, then bias in VSR model is reduced, but computational resources for sample generation increase
Solution Approach 1:
The patent generates synthetic samples for only the most critical under-represented groups rather than attempting to balance all possible groups equally. This partial action approach focuses computational resources on the groups that will provide the greatest improvement in overall model fairness and accuracy.
Solution Approach 2:
The system efficiently generates synthetic samples by applying parameter transformations to existing audio data rather than training complex generative models from scratch. This parameter-based approach reduces computational energy requirements while still producing diverse training samples.
Data Source
AI summary
Systems, methods, and computer-readable media for reducing a bias in visual speech recognition (VSR). In the present embodiments, a comprehensive analysis of the bias (e.g., determining type and severity of the bias) can be performed for each sample in the training data, such as age, gender, and ethnicity, for example. Further, synthetic training data can be generated for under-represented groups using various techniques, such as generative adversarial networks (GANs), for example. Additionally, synthetic video generation can be performed using different modes (e.g., six modes) to ensure quantities and diversity in the synthetic samples. A combination of the real data and the synthetic training data generated can be used to train a VSR model.