Clockwork Hierarchical Variational Autoencoder for Prosody Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing speech synthesis systems struggle to accurately model prosody, resulting in synthesized speech that lacks expressiveness and fails to convey intonation, stress, rhythm, and style effectively.
Innovation Solution
A clockwork hierarchical variational autoencoder (CHiVE) is employed to encode reference audio signals into fixed-length utterance embeddings, which are then decoded to predict syllable durations, pitch, and energy contours, using a hierarchical structure of LSTM cells variably clocked to handle linguistic features and generate synthesized speech with intended prosody.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional concatenative and parametric synthesis models are used, then intelligible speech can be generated, but the synthesized speech lacks naturalness and expressiveness due to ineffective prosody modeling
Solution Approach 1:
The patent segments the speech synthesis model into multiple hierarchical levels (phoneme level, syllable level, utterance level) with separate LSTM networks for each level. This segmentation allows independent optimization of prosody modeling at each linguistic level while maintaining overall system coherence, resolving the contradiction between intelligibility and naturalness.
Solution Approach 2:
The patent introduces a hierarchical temporal dimension by using variably clocked LSTM networks where lower levels (phonemes) operate at faster time scales than higher levels (syllables, utterances). This multi-scale temporal modeling captures prosodic patterns across different time dimensions, improving naturalness without sacrificing intelligibility.
2Adaptability or versatility
If global style tokens are used to model acoustic expressiveness, then a range of prosodic variations can be learned, but the model lacks precise control over syllable-level duration, pitch, and energy contours
Solution Approach 1:
The patent divides the prosody modeling task into separate components at different hierarchical levels: phoneme-level duration prediction, syllable-level pitch contour prediction, and utterance-level style modeling. This segmentation provides precise control over syllable-level parameters while maintaining overall acoustic expressiveness through the hierarchical structure.
Solution Approach 2:
The patent employs dynamically clocked LSTM networks where the sampling rate varies across hierarchical levels. The phoneme-level LSTM operates at a faster rate than the syllable-level LSTM, allowing the model to adaptively capture temporal dynamics at appropriate scales for each prosodic parameter, achieving both versatility and precision.
3Manufacturing precision
If a hierarchical structure with variably clocked LSTM cells is used, then precise prosodic prediction can be achieved, but the device complexity increases
Solution Approach 1:
The patent segments the complex hierarchical model into modular LSTM components, each responsible for a specific linguistic level and prosodic parameter. This modular segmentation makes the complex architecture more manageable and trainable while maintaining high prediction accuracy through coordinated operation of the segmented components.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
A method (400) for representing an intended prosody in synthesized speech (152) includes receiving a text utterance (320) having at least one word (250), and selecting an utterance embedding (260) for the text utterance. Each word in the text utterance has at least one syllable (240) and each syllable has at least one phoneme (230). The utterance embedding represents an intended prosody. For each syllable, using the selected utterance embedding, the method also includes: predicting a duration of the syllable by encoding linguistic features (232) of each phoneme of the syllable with a corresponding prosodic syllable embedding (245) for the syllable; predicting a pitch contour of the syllable based on the predicted duration for the syllable; and generating a plurality of fixed-length predicted pitch frames (280) based on the predicted duration for the syllable. Each fixed-length predicted pitch frame represents part of the predicted pitch contour of the syllable.