Speaker Style Transfer in Text-to-Speech With Limited Reference Audio
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text-to-speech (TTS) models struggle to generate high-quality, personalized speech for new speakers with limited reference data, leading to low adaptability and poor personal style in multi-speaker scenarios.
Innovation Solution
The Adaptive Speech with Oil Water Separation (AS-OWS) method uses an 'oil water separation' model to learn speaker features and a lightweight linear projection, enabling flexible speech synthesis and improved adaptability to new speakers with short reference audio.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional TTS models are used for multi-speaker scenarios, then they can handle multiple speakers, but they fail to generate high-quality personalized speech for new speakers with limited reference data
Solution Approach 1:
The patent segments speech representation into separate style features and content features using style projection networks. The style encoder extracts style features from reference waveforms, while the content encoder processes input text independently. This segmentation allows the model to adapt style features to new speakers without retraining the entire system, resolving the contradiction between adaptability and speech quality.
Solution Approach 2:
The patent changes the parameter representation by projecting style features into a latent space using learnable projection matrices. This parameter transformation enables flexible adaptation to new speakers by simply updating style embeddings rather than retraining the full model, while maintaining high speech quality through the structured separation of style and content parameters.
2Adaptability or versatility
If TTS models are trained with extensive data for multiple speakers, then they improve coverage, but they require large amounts of training data for each new speaker
Solution Approach 1:
The patent extracts style features from reference waveforms using a dedicated style encoder, separating them from content information. This extraction allows the system to learn speaker characteristics from minimal reference data without requiring extensive training corpora for each new speaker, thus improving multi-speaker coverage while reducing data requirements.
Solution Approach 2:
The patent creates a universal TTS framework where a single model can handle multiple speakers through style projection networks. The content encoder and synthesis components remain universal and speaker-agnostic, while only the style embeddings need to be updated for new speakers, enabling multi-speaker coverage without retraining the entire system.
3Manufacturing precision
If TTS models use complex architectures for personalization, then they improve speech quality, but they increase model complexity and training difficulty
Solution Approach 1:
The patent segments the model into distinct components: style encoder, content encoder, style projection network, and synthesis module. This segmentation reduces complexity by allowing each component to be optimized independently and enables flexible combination for different personalization scenarios without redesigning the entire architecture.
Solution Approach 2:
The patent introduces style projection networks as intermediary components that bridge the style encoder and the synthesis module. These projection networks transform style features into a format suitable for the synthesis process, simplifying the overall architecture by providing a clear interface between style adaptation and speech generation components.
Data Source
AI summary
Embodiments of the present disclosure relate to a method, a device, and a computer program product for text to speech. The method includes encoding a reference waveform of a first speaker to obtain an encoded style feature separated from a second speaker. The method further includes transferring the encoded style feature to a spectrogram obtained by encoding an input text, to obtain a style transferred spectrogram. The method further includes converting the style transferred spectrogram into a time-domain speech waveform. According to the method for text to speech in the present disclosure, a comparative learning framework can also flexibly and effectively synthesize speech with a style of a target speaker, thus realizing lightweight speech style transfer, making it possible to learn high-quality and recognizable features of speech synthesis, and realizing effective speaker feature learning. In addition, the model will be beneficial to other downstream tasks.