Neural TTS Phoneme-Grapheme Alignment for Natural Speech
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural text-to-speech (TTS) models fail to preserve the relationship between phonemes and graphemes, leading to issues such as pronunciation ambiguity and alignment challenges, particularly in homophones, which affects the accuracy and naturalness of synthesized speech.
Innovation Solution
The use of an augmented encoder that incorporates both phoneme and grapheme tokens, utilizing a combination of four types of embeddings (token, segment, position, and word position embeddings) to align and represent the relationship between phonemes and graphemes, enabling pre-training on both phonemes and graphemes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If TTS models use phoneme representation only, then the processing speed is improved, but the accuracy of preserving text information deteriorates
Solution Approach 1:
The patent merges phoneme tokens and grapheme tokens into a unified input representation for the TTS model. The encoder receives both phoneme sequences (for pronunciation guidance) and grapheme sequences (for text information preservation), allowing the model to leverage the advantages of both representations simultaneously without sacrificing processing speed or accuracy
Solution Approach 2:
The patent creates a composite input representation that combines two different token types (phonemes and graphemes) with complementary properties. Phonemes provide pronunciation accuracy while graphemes preserve text information, and their combination creates a more robust input that resolves the contradiction between processing efficiency and information preservation
2Measurement precision
If TTS models preserve grapheme information, then the text information accuracy is improved, but the model complexity increases
Solution Approach 1:
The patent segments the input processing into distinct phoneme token handling and grapheme token handling pathways within the encoder. This segmentation allows the model to process different token types through specialized mechanisms while sharing underlying computational resources, reducing the overall complexity increase that would result from processing both representations fully independently
3Reliability
If TTS models use both phoneme and grapheme tokens, then the speech naturalness is improved, but the computational resources required increase
Solution Approach 1:
The patent designs a unified encoder architecture that handles both phoneme and grapheme tokens through shared computational components. The same encoder layers process both token types, allowing the model to leverage grapheme information for improved speech naturalness while avoiding the need for separate dedicated processing pathways that would significantly increase computational resource requirements
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
A method (400) includes receiving a text input (152) including a sequence of words represented as an input encoder embedding (210). The input encoder embedding includes a plurality of tokens (212), with the plurality of tokens including a first set of grapheme tokens (212G) representing the text input as respective graphemes and a second set of phoneme tokens (212P) representing the text input as respective phonemes. The method also includes, for each respective phoneme token of the second set of phoneme tokens: identifying a respective word of the sequence of words corresponding to the respective phoneme token and determining a respective grapheme token representing the respective word of the sequence of words corresponding to the respective phoneme token. The method also includes generating an output encoder embedding (Vc) based on a relationship between each respective phoneme token and the corresponding grapheme token determined to represent a same respective word as the respective phoneme token.