Semi-Supervised Text-to-Speech With Semantic and Acoustic Tokens
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional neural networks for generating speech from text require large amounts of labeled data, particularly parallel data, which is scarce for low-resource languages, leading to limited diversity and speech generation capacity.
Innovation Solution
A method that leverages audio-only data to train generative neural networks by dividing the process into two sequence-to-sequence tasks: generating semantic representations from input text and acoustic representations from semantic representations, using pre-training and backtranslation to reduce the need for parallel data and incorporating voice prompting to enhance diversity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If conventional neural networks are trained using large amounts of labeled parallel data, then speech generation quality is improved, but data collection time and storage requirements increase significantly
Solution Approach 1:
The patent segments the training process into two distinct phases: self-supervised pre-training on unlabeled audio data, and supervised fine-tuning on limited labeled data. This segmentation allows the model to learn general speech representations without requiring extensive labeled parallel data, thereby reducing data collection time while maintaining speech generation quality
Solution Approach 2:
The patent performs self-supervised pre-training on unlabeled audio data before the main supervised training task. This preliminary action enables the model to acquire robust speech representations in advance, reducing the amount of labeled data needed for subsequent fine-tuning and thus decreasing overall data collection time
2Manufacturing precision
If conventional neural networks are trained using large amounts of labeled parallel data, then speech generation quality is improved, but storage requirements increase significantly
Solution Approach 1:
The patent separates the training data into unlabeled audio data (for pre-training) and labeled parallel data (for fine-tuning). This segmentation allows the system to utilize abundant unlabeled data for representation learning while requiring minimal labeled data for task-specific training, thereby reducing overall storage requirements
Solution Approach 2:
The patent employs self-supervised learning where the model learns from unlabeled audio data without human annotation. This self-service approach eliminates the need to store and process large amounts of manually labeled data, reducing storage requirements while still achieving high speech generation quality
3Adaptability or versatility
If large amounts of parallel data are used for training, then speech generation diversity is improved, but processing complexity and computational resources increase
Solution Approach 1:
The patent divides the training process into self-supervised pre-training and supervised fine-tuning stages. During pre-training, the model learns diverse speech patterns from unlabeled data using simplified objectives, reducing processing complexity. The subsequent fine-tuning stage then adapts this diverse knowledge to specific tasks with minimal labeled data
Solution Approach 2:
The patent changes the training objective parameters between pre-training and fine-tuning phases. During pre-training, the model learns general speech representations with diverse acoustic features. During fine-tuning, the parameters are adjusted to focus on task-specific objectives, maintaining diversity while reducing computational complexity
4Loss of time
If conventional neural networks are trained with limited labeled data, then data collection requirements are reduced, but speech generation diversity and capacity are limited
Solution Approach 1:
The patent performs self-supervised pre-training on unlabeled audio data before supervised fine-tuning. This preliminary action enables the model to acquire diverse speech representations from abundant unlabeled data, ensuring speech generation diversity is maintained even when labeled data collection is minimized
Solution Approach 2:
The patent utilizes different learning objectives for pre-training and fine-tuning. During pre-training, the model learns diverse acoustic and linguistic representations from unlabeled data. During fine-tuning, the objectives are changed to task-specific goals while preserving the diversity learned earlier, achieving both reduced data collection requirements and maintained speech generation diversity
Data Source
Figure 1
Figure 2
Figure 3A~3C
AI summary
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an audio signal from input text. In one aspect, a method comprises receiving a request to convert input text into an audio signal, wherein the input text comprises multiple tokenized text inputs, generating, using a first generative neural network, a semantic representation of the tokenized text inputs comprising semantic tokens representing semantic content of the tokenized text inputs, each semantic token being selected from a vocabulary of semantic tokens, generating, using a second generative neural network and conditioned on at least the semantic representation, an acoustic representation of the semantic representation comprising one or more respective acoustic tokens representing acoustic properties of the audio signal, and processing the acoustic representation using a decoder neural network to generate the audio signal.