Expressive Text-to-Speech System Using Convolutional Neural Network
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current text-to-speech systems lack flexibility and controllability in generating expressive speech, particularly in terms of style, speaking rate, and pitch, and are often computationally intensive, making them impractical for devices with limited capabilities.
Innovation Solution
A system that uses a deep convolutional neural network acoustic model conditioned by expression vectors to generate expressive speech, incorporating an expressivity characterisation module and an LPCNet vocoder, allowing for real-time synthesis of customizable speech on low-power devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If deep neural network based SPSS systems are used to generate expressive speech, then speech quality and expressivity are improved, but computational complexity and training time increase significantly
Solution Approach 1:
The patent segments the expressive speech generation process into two independent stages: (1) expression vector extraction from reference audio using an expressivity characterisation module, and (2) speech synthesis using an expressive acoustic model conditioned on these vectors. This segmentation allows the complex DNN to be trained once offline to extract expression vectors, while the actual speech generation uses a simpler conditioned model, reducing real-time computational complexity while maintaining speech quality.
Solution Approach 2:
The patent performs preliminary extraction of expression vectors from reference audio files before the actual speech synthesis process. The expressivity characterisation module pre-processes reference audio to generate expression vectors that capture prosodic information, which are then used to condition the acoustic model. This preliminary action separates the computationally intensive expression extraction from the speech generation, reducing real-time computational requirements.
2Adaptability or versatility
If Tacotron-GST with style tokens is used for expressive TTS, then style transfer capability is improved, but training time increases due to many recurrent layers
Solution Approach 1:
The patent replaces the recurrent neural network architecture (many recurrent layers in Tacotron-GST) with a convolutional neural network-based expressivity characterisation module. This substitution maintains the style transfer capability through expression vectors that condition the acoustic model, while significantly reducing training time by eliminating the computational burden of multiple recurrent layers.
3Manufacturing precision
If neural vocoders like WaveNet are used, then speech quality is improved, but computational requirements become impractical for devices with limited capabilities
Solution Approach 1:
The patent extracts only the essential prosodic information from reference audio into compact expression vectors, separating the expressivity information from the full audio signal. This extraction allows the system to condition the acoustic model with minimal computational overhead while maintaining speech quality, avoiding the need for computationally intensive neural vocoders like WaveNet during real-time generation.
Data Source
AI summary
A system for synthesising expressive speech includes: an interface configured to receive an input text for conversion to speech; a memory; and at least one processor coupled to the memory. The processor is configured to generate, using an expressivity characterisation module, a plurality of expression vectors, wherein each expression vector is a representation of prosodic information in a reference audio style file, and synthesise expressive speech from the input text, using an expressive acoustic model comprising a deep convolutional neural network that is conditioned by at least one of the plurality of expression vectors.