Speech Conversion Personalization With Dynamic Sub-Model Biasing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing speech conversion models struggle with accuracy for users with atypical speech patterns due to their large size and computational inefficiencies, making them difficult to retrain and update, and they often fail to perform well for new phrases or slang.
Innovation Solution
Integrate sub-models with a base speech conversion model to personalize it for specific users or domains, allowing dynamic loading and disabling of these sub-models to enhance accuracy without altering the base model, using residual adaptor layers and multi-headed self-attention mechanisms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a large speech conversion model is trained on large data sets to achieve accurate speech conversion, then the model can convert speech accurately into another form, but the model becomes difficult to retrain and update, and computationally inefficient for users with atypical speech patterns
Solution Approach 1:
The patent divides the speech conversion model into two segments: a base model trained on large datasets for general speech conversion, and specialized sub-models for specific speakers or domains. This segmentation allows the base model to maintain high accuracy while sub-models can be independently trained and updated for specific use cases without retraining the entire system.
Solution Approach 2:
The patent extracts specialized knowledge from the large base model by creating separate sub-models that capture speaker-specific or domain-specific characteristics. This extraction allows the main model to remain computationally efficient while specialized sub-models handle specific cases, reducing the complexity of retraining the entire system.
2Adaptability or versatility
If a generic speech conversion model is used for a large set of users, then the model can handle diverse speech inputs, but it fails to perform well for users with atypical speech patterns
Solution Approach 1:
The patent applies local quality by making different parts of the system specialize in different functions. The base model provides general speech conversion capability for diverse users, while specialized sub-models provide high-quality, accurate conversion for specific speakers or domains with atypical speech patterns. This allows the system to optimize performance locally for different user types.
Solution Approach 2:
The patent implements dynamics by allowing the system to dynamically select and switch between different sub-models based on the input speaker or domain. This dynamic adaptation enables the system to maintain versatility for general use while achieving high precision for specific atypical speech patterns when needed.
3Adaptability or versatility
If the speech conversion model is trained on generalized training data to be speaker-independent, then the model can process various speakers, but it cannot recognize user-specific atypical speech patterns
Solution Approach 1:
The patent segments the modeling approach into a speaker-independent base model trained on generalized data, and speaker-specific sub-models trained on user-specific atypical speech patterns. This segmentation allows the system to maintain speaker independence for general processing while achieving high precision for recognizing atypical speech patterns through specialized sub-models.
Data Source
AI summary
A method for speech conversion includes obtaining a speech conversion model configured to convert input utterances of human speech directly into corresponding output utterances of synthesized speech. The method further includes receiving a speech conversion request including input audio data corresponding to an utterance spoken by a target speaker associated with atypical speech and a speaker identifier uniquely identifying the target speaker. The method includes activating, using the speaker identifier, a particular sub-model for biasing the speech conversion model to recognize a type of the atypical speech associated with the target speaker identified by the speaker identifier. The method includes converting, using the speech conversion model biased by the activated particular sub-model, the input audio data corresponding to the utterance spoken by the target speaker associated with atypical speech into output audio data corresponding to a synthesized canonical fluent speech representation of the utterance spoken by the target speaker.