Multi-Input Speech Synthesis Model for Chinese Pronunciation Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The Tacotron model faces challenges in Chinese speech synthesis due to the large number of Chinese characters, homophones, and differences in pronunciation, leading to poor synthesis effects when using syllables or phonemes as input units, especially with issues related to the pronunciation of finals and homophones.
Innovation Solution
A training method that takes syllable, phoneme, and Chinese character input sequences, fuses their encoded representations, and uses an attention module to generate a weighted average for input to a decoder, improving pronunciation and naturalness by jointly training the encoder and decoder.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If Chinese characters are directly used as input units, then the model can handle all Chinese characters, but the pronunciations of low-frequency Chinese characters cannot be fully learned due to data sparseness
Solution Approach 1:
The patent merges multiple input unit types (Chinese characters, syllables, phonemes) into a unified model framework. The model can selectively use different input units based on the specific task or data availability, combining the comprehensive coverage of character-based input with the pronunciation accuracy of phoneme-based input.
2Measurement precision
If phonemes are used as input units, then homophones are more fully trained by sharing units, but the manner of articulation differences (e.g., functional words vs. real words) cannot be distinguished
Solution Approach 1:
The patent applies local quality by allowing different input unit representations for different contexts. The model can use phoneme-level abstraction where pronunciation consistency is needed, while preserving character or syllable-level information where manner of articulation distinctions are important, creating locally optimized representations throughout the system.
3Device complexity
If syllables are used as input units, then the model structure is simpler, but the pronunciation process for independent finals is insufficient
Solution Approach 1:
The patent segments the syllable into its constituent phonemes, allowing the model to process syllables as composite units of phonemes. This segmentation enables the model to handle independent finals by treating them as phoneme sequences, thereby improving pronunciation completeness while maintaining a relatively simple overall structure through the systematic decomposition of linguistic units.
Data Source
AI summary
The present application discloses a training method and an apparatus for a speech synthesis model, electronic device, and storage medium. The method includes: taking a syllable input sequence, a phoneme input sequence and a Chinese character input sequence of a current sample as inputs of an encoder of a model to be trained, to obtain encoded representations of these three sequences at an output end of the encoder; fusing the encoded representations of these three sequences, to obtain a weighted combination of these three sequences; taking the weighted combination as an input of an attention module, to obtain a weighted average of the weighted combination at each moment at an output end of the attention module; taking the weighted average as an input of a decoder of the model to be trained, to obtain a speech Mel spectrum of the current sample at an output end of the decoder.