Neural TTS Transducer With Iterative Audio Code Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional text-to-speech systems face challenges in aligning source text with phonemes to produce natural-sounding audio, often leading to mispronunciation and jittery speech, and require significant computing resources.
Innovation Solution
A TTS transducer architecture comprising a neural transducer and a residual codebook head (RCH) iteratively generates audio codes to align text with phonemes, eliminating the need for text-to-mel-spectrogram generators and vocoders, thus improving audio quality and reducing computational requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional text-to-speech systems use text-to-mel-spectrogram generators and vocoders to convert text to audio, then speech output can be generated, but the systems require significant computing resources and produce jittery, unnatural-sounding speech with mispronunciations
Solution Approach 1:
The patent extracts and removes the text-to-mel-spectrogram generator and vocoder components from the conventional TTS architecture. By eliminating these complex components, the system reduces computational requirements while maintaining speech generation capability through the transducer model that directly maps text to audio spectrograms.
Solution Approach 2:
The patent merges the functions of text processing, phoneme alignment, and audio generation into a single integrated transducer model. This consolidation eliminates the need for separate text-to-mel-spectrogram generation and vocoding stages, reducing overall system complexity while improving speech quality through unified optimization.
2Measurement precision
If conventional TTS systems use traditional text-to-phoneme mapping methods, then text can be converted to speech, but the alignment between source text and phonemes is inaccurate leading to mispronunciation and skipped text portions
Solution Approach 1:
The transducer model incorporates feedback mechanisms through its encoder-decoder architecture with attention mechanisms. The model continuously refines the alignment between text and phonemes by using feedback from the decoder predictions to adjust the encoder representations, resulting in more accurate text-phoneme mapping and natural speech output without hallucinations or skipped portions.
Data Source
AI summary
Disclosed are apparatuses, systems, and techniques that use a text-to-speech (TTS) transducer to perform TTS operations. The techniques include generating an initial input for a second model using an output of a first model. The techniques include generating, using the second model and the initial input, a first set of audio codes. The techniques include iteratively generating subsequent sets of audio codes using, at each iteration, the second model and a respective subsequent input for the second model. The respective subsequent input can reflect at least one previous set of audio codes generated by the second model.