Iterative Speech Model Training via Voice Conversion Augmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training models for speech recognition, voice conversion, and text-to-speech on low-resource domains often result in overfitting, making them unsuitable for practical applications due to sub-optimal accuracy and reliance on sub-par automatic speech recognition models.
Innovation Solution
An iterative training framework that uses voice conversion models as data augmentation for speech recognition and text-to-speech models, leveraging the encoder of speech recognition models for linguistic content preservation, and fine-tuning these models in a staggered manner to improve accuracy and robustness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If models are trained on low-resource domains, then training efficiency is improved, but model accuracy deteriorates due to overfitting
Solution Approach 1:
The patent creates synthetic speech data by copying and transforming existing speech samples through voice conversion models. This allows the system to generate additional training data without requiring more real-world recordings, thereby maintaining training efficiency while improving model accuracy through data augmentation.
Solution Approach 2:
The patent performs preliminary voice conversion and data augmentation before the main training process. By pre-processing and creating diverse speech samples in advance, the system prepares enriched training data that prevents overfitting during subsequent model training, thus improving accuracy without sacrificing training efficiency.
2Device complexity
If voice conversion model relies on sub-optimal automatic speech recognition model, then system complexity is reduced, but feature extraction quality deteriorates
Solution Approach 1:
The patent implements a feedback mechanism where the voice conversion model's output is evaluated and used to iteratively improve the automatic speech recognition model. This closed-loop approach allows the system to maintain simplicity while progressively enhancing feature extraction quality through repeated refinement cycles.
Solution Approach 2:
The patent makes the system dynamic by allowing the automatic speech recognition model to be updated and retrained based on feedback from the voice conversion model. This dynamic adaptation enables the feature extraction quality to improve over time without permanently increasing system complexity.
3Reliability
If iterative training is performed to improve model accuracy, then model robustness is improved, but computational time increases
Solution Approach 1:
The patent employs periodic training cycles where the models are trained, evaluated, and retrained in structured intervals. This periodic approach allows the system to achieve robustness through iterative improvement while managing computational time by organizing training into efficient, repeating cycles with early stopping criteria.
Solution Approach 2:
The patent applies partial training iterations by performing a limited number of training cycles that are sufficient to achieve adequate robustness without excessive computational expenditure. This approach balances model improvement with time constraints by implementing just enough iterative training to reach acceptable performance levels.
Data Source
AI summary
An electronic device and method for iterative enhancement of speech recognition, voice conversion, and text-to-speech models are disclosed. The electronic device receives a text dataset and a dataset associated with a speech recognition task. Speech recognition, voice conversion, and TTS conversion models are trained for corresponding tasks. The device further iteratively executes operations, which include: generating an augmented speech dataset via application of the trained voice conversion model, finetuning the TTS conversion model using the augmented speech dataset, applying the finetuned TTS conversion model to the text dataset to generate speech samples, applying the trained voice conversion model to the speech samples to create an augmented text-speech dataset, finetuning the speech recognition model utilizing the augmented text-speech dataset, and finetuning the trained voice conversion model using the finetuned speech recognition model. This operations are executed until a loss associated with the voice conversion model falls below a threshold loss.