Neural Text-to-Speech Synthesis via Mel-Spectrogram Decoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network-based text-to-speech systems require significant domain expertise and complex linguistic and acoustic features, making them difficult to produce high-quality speech, and consume more computational resources.
Innovation Solution
A simplified speech synthesis pipeline using a single neural network trained on sample input texts and mel-frequency spectrograms, where a decoder neural network generates mel-frequency spectrograms and a separately-trained vocoder network converts these into time-domain audio waveforms, eliminating the need for linguistic features.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If traditional text-to-speech systems use complex linguistic and acoustic features, then speech quality can be improved, but system complexity and computational resources increase significantly
Solution Approach 1:
The patent extracts and eliminates the complex linguistic feature generation components from the traditional TTS pipeline. By using a neural network that directly maps text to mel-frequency spectrograms without requiring separate linguistic feature extraction, the system removes the burden of domain expertise while maintaining speech quality.
Solution Approach 2:
The neural network is designed to perform multiple functions within a single unified architecture. It simultaneously handles text processing, acoustic feature generation, and spectrogram prediction, replacing the need for multiple specialized components (linguistic feature generators, acoustic models, etc.) that were previously required.
2Manufacturing precision
If traditional text-to-speech systems use complex linguistic and acoustic features, then speech quality can be improved, but computational resource consumption increases
Solution Approach 1:
The patent removes the computationally expensive linguistic feature extraction and manual phoneme generation steps from the pipeline. The direct text-to-spectrogram mapping approach eliminates these intermediate processing stages, significantly reducing computational resource requirements while preserving speech quality.
Solution Approach 2:
The system changes the parameter representation from complex linguistic features requiring domain expertise to mel-frequency spectrograms that can be directly learned by the neural network from text-data pairs. This parameter transformation simplifies the computational task while maintaining the ability to generate high-quality speech.
3Manufacturing precision
If existing systems require robust lexicons and linguistic features, then speech synthesis accuracy can be improved, but ease of operation decreases
Solution Approach 1:
The neural network system is self-sufficient and does not require external linguistic knowledge or manually constructed lexicons. It learns directly from text-spectrogram data pairs, automatically acquiring the necessary linguistic and acoustic knowledge during training. This eliminates the need for operators to possess domain expertise in phonetics or linguistics.
Solution Approach 2:
The system performs preliminary learning during the training phase, where the neural network automatically acquires linguistic patterns and acoustic knowledge from large amounts of text and corresponding spectrogram data. This preliminary action eliminates the need for manual lexicon construction and linguistic feature engineering that would otherwise be required before deployment.
Data Source
AI summary
Methods, systems, and computer program products for generating, from an input character sequence, an output sequence of audio data representing the input character sequence. The output sequence of audio data includes a respective audio output sample for each of a number of time steps. One example method includes, for each of the time steps: generating a mel-frequency spectrogram for the time step by processing a representation of a respective portion of the input character sequence using a decoder neural network; generating a probability distribution over a plurality of possible audio output samples for the time step by processing the mel-frequency spectrogram for the time step using a vocoder neural network; and selecting the audio output sample for the time step from the possible audio output samples in accordance with the probability distribution.