Joint ASR and TTS Training via Adversarial Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current end-to-end deep learning-based Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) systems require paired text and audio samples for training, which are difficult to obtain in significant amounts, limiting their effectiveness.
Innovation Solution
An end-to-end deep-learning-based system that jointly trains ASR and TTS neural networks using unpaired samples through an adversarially-trained approach, allowing the networks to be deployed individually or simultaneously, with a training process involving text and audio encoders/decoders and discriminators to adjust parameters based on calculated losses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If paired text and audio samples are used for training ASR and TTS systems, then the systems can be trained with traditional supervised learning, but the difficulty of obtaining significant amounts of paired samples limits training effectiveness
Solution Approach 1:
The patent divides the training process into separate stages: first training ASR on audio samples, then training TTS on text samples, and finally combining them through adversarial training. This segmentation allows each component to be trained independently on unpaired data before integration, resolving the contradiction by eliminating the need for paired samples while maintaining training effectiveness
Solution Approach 2:
The patent introduces a discriminator as an intermediary component that receives outputs from both ASR and TTS systems. The discriminator evaluates the consistency between text and audio representations, enabling the system to train on unpaired samples by mediating the interaction between the two modalities and ensuring coherent alignment without requiring direct paired data
2Ease of manufacture
If unpaired samples are used for training, then the ease of data collection is improved, but the system complexity increases due to the need for adversarial training and multiple neural networks
Solution Approach 1:
The patent creates a universal training framework where the same adversarial neural network structure can handle both ASR and TTS tasks. The discriminator serves multiple functions by evaluating both audio-to-text and text-to-audio transformations, allowing a single system architecture to address both conversion directions while maintaining manageable complexity through shared components
Solution Approach 2:
The patent implements feedback mechanisms where the discriminator's evaluations are used to adjust the parameters of both ASR and TTS neural networks. This feedback loop enables the system to iteratively improve the consistency between text and audio representations, managing the complexity through structured parameter adjustment based on discriminator signals rather than requiring complex manual tuning
Data Source
AI summary
An end-to-end deep-learning-based system that can solve both ASR and TTS problems jointly using unpaired text and audio samples is disclosed herein. An adversarially-trained approach is used to generate a more robust independent TTS neural network and an ASR neural network that can be deployed individually or simultaneously. The process for training the neural networks includes generating an audio sample from a text sample using the TTS neural network, then feeding the generated audio sample into the ASR neural network to regenerate the text. The difference between the regenerated text and the original text is used as a first loss for training the neural networks. A similar process is used for an audio sample. The difference between the regenerated audio and the original audio is used as a second loss. Text and audio discriminators are similarly used on the output of the neural network to generate additional losses for training.