Acoustic Model Emotion Embedding for Natural Speech Synthesis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing speech synthesis technologies produce unnatural and emotionally blunt speech, lacking rich emotional expressions.
Innovation Solution
A method and system for training an acoustic model that incorporates an encoder, emotion embedding vector determination module, and an auxiliary module with unsupervised and supervised components to generate a comprehensive emotion vector, enabling the creation of a prediction mel spectrum and speech that reflects emotional nuances.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If traditional speech synthesis technology is used, then the synthesis process is simple, but the synthesized speech is blunt and unnatural lacking emotional expressions
Solution Approach 1:
The acoustic model is segmented into multiple specialized modules: an encoder for text processing, an emotion embedding vector determination module for emotion representation, and a decoder for mel spectrum generation. This segmentation allows each module to specialize in specific tasks, improving overall naturalness while managing complexity through modular design.
Solution Approach 2:
The patent introduces a new dimension of emotion embedding vectors alongside traditional text encoding. By combining text sequence vectors with emotion embedding vectors in a cascade structure, the model adds an emotional dimension to speech synthesis, transforming blunt output into emotionally rich speech without overwhelming the base architecture.
2Manufacturing precision
If emotion embedding vectors are added to the acoustic model, then emotional expression is improved, but training complexity increases
Solution Approach 1:
An auxiliary module is introduced as an intermediary between the emotion embedding vector determination module and the decoder. This auxiliary module processes and refines the emotion information before it reaches the decoder, simplifying the training process by providing a dedicated layer for emotion processing and reducing the burden on other components.
Solution Approach 2:
The training process incorporates feedback mechanisms where the auxiliary module and emotion embedding vectors are iteratively adjusted based on loss targets. This feedback loop allows the model to learn optimal emotion representations through iterative refinement, managing training complexity through systematic optimization rather than one-time complex processing.
3Manufacturing precision
If multiple loss targets are used for training, then speech naturalness and emotion accuracy are improved, but training time increases
Solution Approach 1:
The auxiliary module performs preliminary processing of emotion information during training, pre-computing and refining emotion embedding vectors before they are used in the final speech generation. This preliminary action allows the main training loop to focus on optimizing the core speech synthesis path, reducing overall training time while maintaining high emotion accuracy through the pre-processed emotion representations.
Data Source
AI summary
The present disclosure provides acoustic model training methods and systems, and speech synthesis methods and systems. An acoustic model training method may include obtaining a plurality of training samples. Each of the plurality of training samples may include a sample text input, a sample emotion label corresponding to the sample text input, and a sample reference mel spectrum corresponding to the sample text input. The acoustic model training method may include inputting the plurality of training samples into a target model. The target model may include the acoustic model and an auxiliary module. The acoustic model training method may further include iteratively adjusting at least one model parameter of the acoustic model based on a loss target.