Neural Vocoder Multi-Threading for Low-Latency TTS Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing Text-To-Speech (TTS) systems face challenges in achieving natural sounding output with low latency, as previous attempts to reduce inference time often deteriorate audio quality.
Innovation Solution
Implementing a multi-threading approach that optimally utilizes multiple threads at the neural vocoder level of the TTS processing pipeline, while performing preprocessing and acoustic model operations serially, to generate speech waveforms in parallel.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If serial processing is used in TTS pipeline, then audio quality is maintained, but inference time latency is high
Solution Approach 1:
The patent divides the sound frequency data (Mel spectrogram) into multiple segments and processes each segment independently through separate neural network instances in parallel. This segmentation allows the system to maintain audio quality by preserving the integrity of each segment while reducing overall inference time through concurrent processing of multiple segments.
2Loss of time
If parallel processing is used to reduce latency, then inference time is reduced, but audio quality deteriorates
Solution Approach 1:
The patent applies partial parallelization by processing only the sound frequency data generation and waveform synthesis in parallel through multiple threads, while keeping the preprocessing stage serial. This selective parallelization achieves latency reduction without over-parallelizing the entire pipeline, thereby maintaining audio quality.
3Productivity
If multiple threads are used for processing, then processing speed increases, but system complexity increases
Solution Approach 1:
The patent segments the processing workload into distinct stages (preprocessing, sound frequency data generation, waveform synthesis) and applies multi-threading only to the beneficial stages. This targeted segmentation increases processing speed while minimizing the added complexity by avoiding unnecessary parallelization of the entire system.
4Use of energy by moving object
If preprocessing is performed serially, then resource utilization is optimized, but overall processing time increases
Solution Approach 1:
The patent performs preprocessing serially as a preliminary action that prepares the input text and sound units before the parallel processing stage. This preliminary serial processing optimizes resource utilization by preparing data in an organized manner, while the subsequent parallel stages compensate for the time taken, achieving overall efficiency.
Data Source
AI summary
Techniques discussed herein relate to reducing latency in a Text-To-Speech processing pipeline. A request may be received requesting a speech waveform corresponding to input text provided in the request. The input text may be processed using a set of text preprocessing operations to generate a set of sound units. The set of sound units may be provided to an acoustic model to generate sound frequency data which may be divided into a number of smaller sound frequency data segments corresponding to the number of available computing threads. Each thread may be configured to provide a respective sound frequency data segment to a neural network as input to generate a plurality of speech waveforms. The plurality of speech waveforms may be combined to generate the speech waveform requested. The combined speech waveform may be provided in response to the request.