TTS Adapter Layers for New Speaker Customization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional text-to-speech (TTS) machine learning models require significant amounts of speech data and computational resources to customize for new speakers, and re-training can degrade performance for original speakers.
Innovation Solution
Inserting adapters, or additional network layers, into a TTS model and updating their parameters while freezing the base model's parameters, allows customization for new speakers with reduced data and computational requirements, maintaining performance for original speakers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the TTS machine learning model is re-trained to generate speech for a new speaker, then the model can be customized for the new speaker, but a large amount of speech data and computational resources are required
Solution Approach 1:
The patent segments the TTS model into a base model and adapter components. The base model contains the general TTS capabilities trained on original speakers, while the adapter is a separate, smaller component that captures speaker-specific characteristics. This segmentation allows the adapter to be trained independently with minimal data, avoiding the need to re-train the entire base model.
Solution Approach 2:
The patent extracts the speaker-specific information processing function from the base TTS model and places it in a separate adapter module. This extraction allows the adapter to be trained using only the new speaker's data without requiring the base model to be re-trained, thus reducing the quantity of speech data needed while maintaining customization capability.
2Adaptability or versatility
If the TTS machine learning model is re-trained to generate speech for a new speaker, then the model can be customized for the new speaker, but the performance for previous speakers is degraded
Solution Approach 1:
The patent divides the model into a frozen base model that maintains original speaker performance and a trainable adapter that handles new speaker customization. By segmenting the functionality this way, the base model's parameters remain fixed and preserve original performance, while only the lightweight adapter adapts to new speakers.
Solution Approach 2:
The base model is pre-trained and frozen before introducing the adapter. This preliminary action establishes the foundation for original speaker performance, and subsequent adapter training builds upon this fixed foundation without disturbing the original performance characteristics.
3Adaptability or versatility
If the TTS machine learning model is re-trained to generate speech for a new speaker, then the model can be customized for the new speaker, but substantial time and compute resources are needed
Solution Approach 1:
The patent extracts the heavy computational burden of speaker-specific training from the base model by creating a separate adapter. This extraction allows the adapter to be trained independently and quickly using only new speaker data, eliminating the need to re-train the entire base model and significantly reducing training time.
Solution Approach 2:
The patent changes the training parameters by freezing the base model parameters and only training the adapter parameters. This parameter change transforms the training task from training the entire large model to training only the small adapter, dramatically reducing compute resources and training time required.
4Adaptability or versatility
If the TTS machine learning model is re-trained to generate speech for a new speaker, then the model can be customized for the new speaker, but substantial compute resources are required
Solution Approach 1:
The patent segments the computational workload into a fixed base model that requires no compute resources during adaptation and a small adapter that requires minimal compute resources. This segmentation reduces the overall compute resources needed for customization from training the entire base model to training only the small adapter.
Solution Approach 2:
The patent changes the compute resource requirement by freezing base model parameters and training only the adapter parameters. This parameter change reduces the number of parameters that need to be updated during training, thereby substantially reducing the compute resources and energy consumption required for speaker customization.
Data Source
AI summary
In various examples, one or more text-to-speech machine learning models may be customized or adapted to accommodate new or additional speakers or speaker voices without requiring a full re-training of the models. For example, a base model may be trained on a set of one or more speakers and, after training or deployment, the model may be adapted to support one or more other speakers. To do this, one or more additional layers (e.g., adapter layers) may be added to the model, and the model may be re-trained or updated—e.g., by freezing parameters of the base model while updating parameters of the adapter layers—to generate an adapted model that can support the one or more original speakers of the base model in addition to the one or more additional speakers corresponding to the adapter layers.