Text-to-Speech Duration Prediction Neural Network
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text-to-speech systems using autoregressive neural networks face issues with robustness due to ad-hoc stopping mechanisms, leading to early cut-off or failure to stop, and lack constraints to prevent repetition or skipping of output tokens, resulting in suboptimal audio synthesis.
Innovation Solution
A neural network system that incorporates a duration prediction neural network to predict the duration of each text element, allowing for precise control of audio output length and using a distribution-based approach to model token influence, thereby eliminating early cut-off and minimizing repetition or skipping, and enabling unsupervised training without ground-truth durations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If autoregressive neural networks are used to generate audio outputs iteratively, then the system can generate speech sequentially, but the ad-hoc stopping mechanism causes early cut-off or failure to stop, reducing reliability
Solution Approach 1:
The duration prediction neural network predicts the duration of each text element in advance, before the audio generation process begins. This preliminary duration information is used to control the stopping mechanism, ensuring that the network generates the exact number of audio tokens needed without early cut-off or failure to stop, thereby resolving the reliability issue while maintaining iterative generation capabilities
2Adaptability or versatility
If autoregressive neural networks generate output tokens without constraints, then the generation process is flexible, but repetition or skipping of tokens occurs, reducing manufacturing precision
Solution Approach 1:
The duration prediction neural network provides feedback information about the expected duration of each text element. This feedback is used to guide the audio token generation process, ensuring that the generated audio sequence accurately corresponds to the input text sequence without repetition or skipping, while maintaining the flexibility of the autoregressive generation process
3Ease of manufacture
If conventional non-autoregressive systems upsample input sequence by simple repetition, then the process is simple, but the quality of intermediate sequence is poor, reducing manufacturing precision
Solution Approach 1:
Instead of simple repetition, the system uses the predicted duration values as parameters to control the upsampling process. Each text element is repeated a number of times proportional to its predicted duration, creating a more accurate intermediate sequence that better represents the temporal structure of the input text, thereby improving quality while maintaining computational efficiency
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, synthesizing audio data from text data using duration prediction. One of the methods includes processing an input text sequence that includes a respective text element at each of multiple input time steps using a first neural network to generate a modified input sequence comprising, for each input time step, a representation of the corresponding text element in the input text sequence; processing the modified input sequence using a second neural network to generate, for each input time step, a predicted duration of the corresponding text element in the output audio sequence; upsampling the modified input sequence according to the predicted durations to generate an intermediate sequence comprising a respective intermediate element at each of a plurality of intermediate time steps; and generating an output audio sequence using the intermediate sequence.