Neural Speech Gap Filling for More Natural Text-to-Speech
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing parametric text-to-speech synthesis systems produce speech that does not sound as lively as natural speech due to statistical averaging in training, resulting in gaps in pitch, amplitude, and duration, leading to lower mean opinion scores (MOS) and inefficiencies in data footprint.
Innovation Solution
A neural network-based speech gap filling model is trained to compare synthetic speech data with recorded reference data, extracting features to adjust properties like pitch, amplitude, and duration, aligning them with natural speech, using connectionist temporal classification (CTC) and long short-term memory (LSTM) to improve synthesis quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If statistical averaging is used in TTS training, then training efficiency is improved, but speech quality becomes less lively and natural
Solution Approach 1:
The patent segments the training process into two distinct phases: pre-training with statistical averaging for efficiency, and fine-tuning with individual speech samples for quality. This segmentation allows each phase to optimize for its specific goal without compromising the other.
Solution Approach 2:
The pre-training phase performs preliminary learning using statistical averaging to establish baseline speech patterns. This preliminary action prepares the model for subsequent fine-tuning, where natural speech characteristics are refined using individual samples.
2Manufacturing precision
If more data is stored to improve speech quality, then MOS score is improved, but data footprint increases
Solution Approach 1:
The patent extracts only the most essential individual speech samples needed for fine-tuning, rather than storing comprehensive datasets. This extraction approach achieves high MOS scores by focusing on quality-critical samples while minimizing data footprint.
Solution Approach 2:
The patent changes the training parameter from comprehensive statistical averaging to selective fine-tuning with curated samples. This parameter change achieves high speech quality with reduced data requirements by optimizing for natural speech characteristics.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method, computer program product, and computer system for text-to-speech synthesis is disclosed. Synthetic speech data for an input text may be generated. The synthetic speech data may be compared to recorded reference speech data corresponding to the input text. Based on, at least in part, the comparison of the synthetic speech data to the recorded reference speech data, at least one feature indicative of at least one difference between the synthetic speech data and the recorded reference speech data may be extracted. A speech gap filling model may be generated based on, at least in part, the at least one feature extracted. A speech output may be generated based on, at least in part, the speech gap filling model.