Speech Synthesis Semantic Token Pipeline for Low-Data Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing speech synthesis systems require a high amount of labeled data for training, leading to limitations in accuracy due to excessive feature spans from text to acoustic tokens, affecting the overall synthesis accuracy.
Innovation Solution
A two-stage process involving feature extraction on both prompt audio and input text to obtain semantic and acoustic tokens, followed by predicting the final acoustic token using these tokens, reducing the feature span and requiring less labeled data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If acoustic token is directly predicted from text and prompt audio, then speech synthesis can be performed, but the feature span is excessively large leading to high requirement for labeled data and limiting accuracy
Solution Approach 1:
The patent segments the direct prediction process into two stages: first predicting semantic tokens from text, then predicting acoustic tokens from both the text and prompt audio. This segmentation reduces the feature span complexity by breaking down the excessive direct mapping into manageable intermediate steps, where semantic tokens serve as a bridge between text and acoustic features.
Solution Approach 2:
The patent introduces semantic tokens as an intermediary element between text and acoustic tokens. The semantic token extraction model processes text to generate semantic tokens, which then serve as intermediate features for the acoustic token prediction. This intermediary approach reduces the direct feature span complexity and lowers the requirement for labeled data while maintaining or improving accuracy.
2Measurement precision
If acoustic token is directly predicted from text and prompt audio, then speech synthesis can be performed, but high requirement for labeled data limits accuracy
Solution Approach 1:
The patent divides the training data requirement into two separate models: a semantic token extraction model trained on text-sematic token pairs, and an acoustic token prediction model trained on combined features. This segmentation allows each model to learn from more focused, smaller datasets rather than requiring a single large labeled dataset for the entire pipeline.
Solution Approach 2:
By introducing semantic tokens as intermediaries, the patent reduces the direct dependency between text and acoustic tokens, thereby reducing the requirement for extensively labeled data. The semantic token extraction model can be trained with less labeled data since it only needs to map text to semantic representations, and the acoustic token model benefits from the structured intermediate features.
3Measurement precision
If feature extraction is performed on both prompt audio and input text to obtain semantic and acoustic tokens, then the feature span is reduced, but the processing complexity increases
Solution Approach 1:
The patent segments the processing pipeline into distinct modules: a semantic token extraction model that processes text, and an acoustic token prediction model that processes both text and prompt audio features. This segmentation organizes the complexity into manageable, independent components that can be trained and optimized separately, reducing the overall processing complexity while maintaining accuracy.
Solution Approach 2:
The patent performs preliminary feature extraction on the prompt audio to obtain acoustic features before combining them with text features for acoustic token prediction. This preliminary action prepares the data in advance, organizing features in a structured manner that simplifies the subsequent prediction process and reduces the complexity of handling raw, unprocessed features.
Data Source
AI summary
Speech synthesis techniques are described herein, which relate to the field of artificial intelligence (AI). The techniques may include performing feature extraction on prompt audio to obtain a prompt semantic token and a prompt acoustic token; performing feature extraction on input text to obtain an input semantic token; acquiring an input acoustic token based on the prompt semantic token, the prompt acoustic token, and the input semantic token; and generating an output audio of the input text based on the input acoustic token. According to this application accuracy of speech synthesis can be improved.