Emotion Speech Synthesis Using Decoupled Timbre and Emotion Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for synthesizing multi-timbre and multi-emotion speech require extensive data collection, increase complexity, and negatively affect speech quality due to varying emotion expression capabilities of speakers.
Innovation Solution
A synthesis method that determines speech features based on target emotion and timbre, allowing synthesis without collecting data for each timbre, using a mapping process to integrate syllable features and timbre information, reducing data collection costs and improving speech quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If speech data of different emotions for each timbre is collected, then multi-timbre and multi-emotion speech synthesis capability is achieved, but data collection cost and system complexity greatly increase
Solution Approach 1:
The patent segments the speech synthesis task into two independent parts: timbre representation (using speaker embeddings from pre-trained models) and emotion representation (using separate emotion classification and generation modules). This segmentation allows the system to handle multiple timbres and emotions without collecting data for all combinations, as each dimension can be processed independently and combined during synthesis.
Solution Approach 2:
The patent introduces an intermediary emotion representation layer that decouples the relationship between timbre and emotion. Instead of directly mapping timbre to emotion-specific speech data, the system uses an intermediate emotion embedding space where emotion characteristics are represented separately from speaker characteristics. This intermediary representation enables flexible combination of different timbres and emotions without requiring extensive paired data collection.
2Adaptability or versatility
If speech data from multiple speakers with different emotion expression capabilities is collected, then coverage of various emotions is improved, but speech quality varies due to inconsistent emotion expression capabilities
Solution Approach 1:
The patent applies local quality by allowing different speakers to contribute to different emotion types based on their strengths. The system selectively weights or selects speaker embeddings based on the target emotion, so that emotions are synthesized using data from speakers who excel at expressing that particular emotion. This ensures high quality for each emotion type while maintaining overall emotion coverage across multiple speakers.
Solution Approach 2:
The patent dynamically adjusts the parameters of speaker embeddings and emotion weights based on the target emotion and desired speech quality. By changing the weighting parameters in the synthesis equation according to the specific emotion being generated, the system optimizes quality for each emotion type while maintaining versatility across multiple emotions and speakers.
3Adaptability or versatility
If extensive speech data collection is performed, then synthesis coverage of multiple timbres and emotions is achieved, but synthesis efficiency and speed decrease
Solution Approach 1:
The patent performs preliminary actions by pre-training speaker embedding models and emotion classification models separately before the actual synthesis task. Speaker embeddings are extracted in advance from audio samples, and emotion representations are pre-computed from text or audio inputs. This preliminary processing separates the computationally intensive data collection and feature extraction phases from the real-time synthesis phase, thereby improving synthesis efficiency while maintaining comprehensive coverage.
Data Source
AI summary
A synthesis method for emotion speech and a synthesis apparatus are provided; the synthesis method includes: determining, by a terminal, an original text, a target emotion, a target timbre, and an emotion timbre corresponding to a to-be-synthesized emotion speech; splitting the original text to obtain a plurality of target syllables constituting the original text; performing mapping on each of the plurality of target syllables based on the target emotion and the emotion timbre to obtain a phonetics speech feature sequence of each target syllable; integrating, by the terminal, the phonetics speech feature sequence of each of the target syllables to obtain a speech feature corresponding to the original text; and synthesizing, by the terminal, the target emotion speech, by using the speech feature and the target timbre.