Multi-scale spectrogram text-to-speech prosody modeling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current text-to-speech systems face challenges in producing speech with appropriate prosody due to the lack of prosodic variations, resulting in synthetic speech that sounds flat and requires more cognitive effort to process, mainly because they predict finer acoustic representations directly from discrete textual inputs without capturing short and long-range dependencies effectively.
Innovation Solution
The implementation of multi-scale spectrogram modeling, which predicts mel spectrograms sequentially from coarser to finer scales, capturing higher-level representations of speech and conditioning each subsequent scale on the previous one to produce prosody appropriate at different linguistic units like sentence, word, and phonemes, thereby improving the naturalness of synthetic speech.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If current TTS systems directly predict spectrograms from discrete textual inputs, then the system complexity is reduced, but the ability to capture prosodic variations and short/long-range dependencies deteriorates
Solution Approach 1:
The patent segments the spectrogram prediction task into multiple scales: a first scale capturing short-range dependencies and a second scale capturing long-range dependencies. This segmentation allows the model to process different temporal contexts separately and combine them, preserving prosodic information that would be lost in a direct prediction approach.
Solution Approach 2:
The patent introduces a temporal scale dimension by predicting spectrograms at multiple time scales rather than a single scale. This transforms the problem from a single-dimensional direct prediction to a multi-dimensional hierarchical prediction, enabling the model to capture both short and long-range prosodic variations.
2Ease of manufacture
If TTS systems use discrete textual tokens with coarse time resolution, then the input processing is simplified, but the ability to model continuous domain spectrograms with fine time resolution deteriorates
Solution Approach 1:
The patent applies preliminary upsampling to the discrete textual tokens before spectrogram prediction. This preliminary action transforms the coarse-resolution input into a finer-resolution representation that better matches the target spectrogram time resolution, enabling more precise temporal modeling while keeping the input processing relatively simple.
3Productivity
If TTS systems predict finer acoustic representations directly from coarser textual inputs, then the processing steps are reduced, but the naturalness of synthetic speech deteriorates
Solution Approach 1:
The patent segments the acoustic representation prediction into multiple scales where a first scale captures short-range dependencies and a second scale captures long-range dependencies. This segmentation allows the model to progressively build up prosodic information from local to global contexts, improving speech naturalness without excessive processing complexity.
Solution Approach 2:
The patent implements a nested hierarchical structure where spectrogram predictions at different scales are combined. The finer-scale predictions are conditioned on coarser-scale predictions, creating a nested structure that progressively refines the acoustic representation and improves naturalness while maintaining processing efficiency.
Data Source
AI summary
Techniques for performing text-to-speech are described. An exemplary method includes receiving a request to generate audio from input text; generating audio from the input text by: generating a first number of vectors from phoneme embeddings representing the input text, predicting one or more spectrograms having the first number of frames using multiple scales wherein a coarser scale influences a finer scale, concatenating the first number of vectors and the predicted one or more spectrograms, generating at least one mel spectrogram from the concatenated vectors and the predicted one or more spectrograms, and converting, with a vocoder, the at least one mel spectrogram frames to audio; and outputting the generated audio according to the request.